[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:47:37 dist-test-slave-j968 krb5kdc[18825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:47:37 dist-test-slave-j968 krb5kdc[18825](info): set up 2 sockets
Sep 28 14:47:37 dist-test-slave-j968 krb5kdc[18825](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 14:47:38.416988 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:47:38 dist-test-slave-j968 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:47:38.536145 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:33987
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:33987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:38.617089 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:38.617336 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:38.617372 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:38.618517 18838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:47:38.618584 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:38.618607 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:38.618628 18838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:47:38.618659 18838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:47:38.620168 18838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:33987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:33987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:38.620471 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:38.620698 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:38.626219 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:38.626245 18838 server_base.cc:1034] running on GCE node
W20260928 14:47:38.626459 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:38.626309 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:38.626793 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:38.627097 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:38.628257 18838 hybrid_clock.cc:648] HybridClock initialized: now 1790606858628246 us; error 44 us; skew 500 ppm
Sep 28 14:47:38 dist-test-slave-j968 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:38.630692 18838 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:47:38.631135 18838 webserver.cc:460] Webserver started at http://127.18.96.190:43539/ using document root <none> and password file <none>
I20260928 14:47:38.631353 18838 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:38.631397 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:38.631560 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:38.632773 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "295afc255b444e3b8c83e4219d087eb9"
format_stamp: "Formatted at 2026-09-28 14:47:38 on dist-test-slave-j968"
server_key: "80bda82798fa1981894218c2e39663da"
server_key_iv: "44495edad320fb753e3dc0885ce464fb"
server_key_version: "encryptionkey@0"
I20260928 14:47:38.633128 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "295afc255b444e3b8c83e4219d087eb9"
format_stamp: "Formatted at 2026-09-28 14:47:38 on dist-test-slave-j968"
server_key: "80bda82798fa1981894218c2e39663da"
server_key_iv: "44495edad320fb753e3dc0885ce464fb"
server_key_version: "encryptionkey@0"
I20260928 14:47:38.634447 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:47:38.635192 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:38.635396 18838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:38.635496 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "295afc255b444e3b8c83e4219d087eb9"
format_stamp: "Formatted at 2026-09-28 14:47:38 on dist-test-slave-j968"
server_key: "80bda82798fa1981894218c2e39663da"
server_key_iv: "44495edad320fb753e3dc0885ce464fb"
server_key_version: "encryptionkey@0"
I20260928 14:47:38.635589 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:38.653478 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:38.656081 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:38.656270 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:38.660914 18838 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:33987
I20260928 14:47:38.660945 18905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:33987 every 8 connection(s)
I20260928 14:47:38.661324 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:47:38.662276 18906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:38.664841 18906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9: Bootstrap starting.
I20260928 14:47:38.665439 18906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:38.665684 18906 log.cc:824] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:38.666426 18906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9: No bootstrap required, opened a new log
I20260928 14:47:38.668079 18906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295afc255b444e3b8c83e4219d087eb9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33987 } }
I20260928 14:47:38.668221 18906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:38.668243 18906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295afc255b444e3b8c83e4219d087eb9, State: Initialized, Role: FOLLOWER
I20260928 14:47:38.668347 18906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295afc255b444e3b8c83e4219d087eb9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33987 } }
I20260928 14:47:38.668430 18906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:47:38.668464 18906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:47:38.668533 18906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:38.669102 18906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295afc255b444e3b8c83e4219d087eb9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33987 } }
I20260928 14:47:38.669224 18906 leader_election.cc:302] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 295afc255b444e3b8c83e4219d087eb9; no voters:
I20260928 14:47:38.669412 18906 leader_election.cc:288] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:47:38.669472 18909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:38.669664 18909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [term 1 LEADER]: Becoming Leader. State: Replica: 295afc255b444e3b8c83e4219d087eb9, State: Running, Role: LEADER
I20260928 14:47:38.669674 18906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:47:38.669775 18909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295afc255b444e3b8c83e4219d087eb9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33987 } }
I20260928 14:47:38.670409 18911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 295afc255b444e3b8c83e4219d087eb9. Latest consensus state: current_term: 1 leader_uuid: "295afc255b444e3b8c83e4219d087eb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295afc255b444e3b8c83e4219d087eb9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33987 } } }
I20260928 14:47:38.670519 18911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:38.670680 18910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "295afc255b444e3b8c83e4219d087eb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295afc255b444e3b8c83e4219d087eb9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33987 } } }
I20260928 14:47:38.670765 18910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:38.671365 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 18838
I20260928 14:47:38.671470 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:47:38.670940 18918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:47:38.672209 18818 external_mini_cluster.cc:1428] Setting key aa97820db2d033aba36832e8c9bc49f0
I20260928 14:47:38.672452 18918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:47:38.676450 18918 catalog_manager.cc:1347] Generated new cluster ID: 9cf1f9dfe95c49d0a17358ec4b983877
I20260928 14:47:38.676517 18918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:47:38 dist-test-slave-j968 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:38.688938 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:38.673106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.18.96.190:33987)
0928 14:47:38.673439 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:38.673447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:38.673774 (+ 327us) server_negotiation.cc:407] Connection header received
0928 14:47:38.676079 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:38.676085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:38.676154 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:38.676316 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:38.678046 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:38.679168 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:38.680447 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:38.680906 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:38.681059 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:38.684846 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:38.684850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:38.684851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:38.684871 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:38.686723 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:38.687342 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:38.687345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:38.687346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:38.687442 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:38.687751 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:38.687752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:38.687753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:38.688070 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:38.688244 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:38.688530 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 14:47:38.688672 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":81,"threads_started":1}
I20260928 14:47:38.693523 18918 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:47:38.694037 18918 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:47:38.701231 18918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 295afc255b444e3b8c83e4219d087eb9: Generated new TSK 0
I20260928 14:47:38.701459 18918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:47:38.732808 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33987
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:38.820493 18932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:38.820698 18932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:38.820720 18932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:38.821867 18932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:38.821910 18932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:38.821938 18932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:47:38.823390 18932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:38.823640 18932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:38.823915 18932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:38.829002 18938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:38.829010 18937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:38.829061 18932 server_base.cc:1034] running on GCE node
W20260928 14:47:38.829002 18940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:38.829387 18932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:38.829586 18932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:38.830724 18932 hybrid_clock.cc:648] HybridClock initialized: now 1790606858830703 us; error 36 us; skew 500 ppm
Sep 28 14:47:38 dist-test-slave-j968 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:38.833289 18932 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:47:38.833738 18932 webserver.cc:460] Webserver started at http://127.18.96.129:43525/ using document root <none> and password file <none>
I20260928 14:47:38.833977 18932 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:38.834023 18932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:38.834164 18932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:38.835229 18932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "59586e98d41a4f14b02757b1dc858fb6"
format_stamp: "Formatted at 2026-09-28 14:47:38 on dist-test-slave-j968"
server_key: "392c6b89fc57434219236e5934fe4bdb"
server_key_iv: "89ffa19149ab81ef6dae7df17bda3b14"
server_key_version: "encryptionkey@0"
I20260928 14:47:38.835583 18932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "59586e98d41a4f14b02757b1dc858fb6"
format_stamp: "Formatted at 2026-09-28 14:47:38 on dist-test-slave-j968"
server_key: "392c6b89fc57434219236e5934fe4bdb"
server_key_iv: "89ffa19149ab81ef6dae7df17bda3b14"
server_key_version: "encryptionkey@0"
I20260928 14:47:38.837033 18932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:47:38.837932 18947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:38.838104 18932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:38.838187 18932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "59586e98d41a4f14b02757b1dc858fb6"
format_stamp: "Formatted at 2026-09-28 14:47:38 on dist-test-slave-j968"
server_key: "392c6b89fc57434219236e5934fe4bdb"
server_key_iv: "89ffa19149ab81ef6dae7df17bda3b14"
server_key_version: "encryptionkey@0"
I20260928 14:47:38.838240 18932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:38.848613 18932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:38.850374 18932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:38.850485 18932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:38.850654 18932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:38.850973 18932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:38.851006 18932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:38.851029 18932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:38.851075 18932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:38.856683 18932 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:40297
I20260928 14:47:38.856757 19060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:40297 every 8 connection(s)
I20260928 14:47:38.857089 18932 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:47:38.858071 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 18932
I20260928 14:47:38.858191 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:47:38.858354 18818 external_mini_cluster.cc:1428] Setting key 130641a3d67d6968330944731ed461f1
Sep 28 14:47:38 dist-test-slave-j968 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
I20260928 14:47:38.867570 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:38.857838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:40023 (local address 127.18.96.190:33987)
0928 14:47:38.857888 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:38.857890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:38.858107 (+ 217us) server_negotiation.cc:407] Connection header received
0928 14:47:38.858761 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:38.858762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:38.858773 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:38.858841 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:38.859889 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:38.860604 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:38.861585 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:38.861953 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:38.862025 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:38.864580 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:38.864582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:38.864582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:38.864591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:38.866260 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:38.866666 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:38.866667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:38.866668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:38.866725 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:38.866958 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:38.866959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:38.866959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:38.867109 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:38.867186 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:38.867429 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 14:47:38.867499 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:47:38.867741 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:38.857797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:33987 (local address 127.18.96.129:40023)
0928 14:47:38.857968 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:38.858010 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:38.858654 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:38.858899 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:38.858903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:38.859558 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:38.859759 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:38.859764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:38.860696 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:38.860697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:38.861496 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:38.861498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:38.862046 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:38.862047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:38.862161 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:38.862742 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:38.862751 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:38.864474 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:38.866354 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:38.866361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:38.866362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:38.866587 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:38.866794 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:38.866795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:38.866795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:38.866890 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:38.867181 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:38.867186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:38.867305 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:47:38.867393 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:47:38.867537 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260928 14:47:38.868332 19061 heartbeater.cc:347] Connected to a master server at 127.18.96.190:33987
I20260928 14:47:38.868439 19061 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:38.868623 19061 heartbeater.cc:511] Master 127.18.96.190:33987 requested a full tablet report, sending...
I20260928 14:47:38.869223 18870 ts_manager.cc:194] Registered new tserver with Master: 59586e98d41a4f14b02757b1dc858fb6 (127.18.96.129:40297)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
I20260928 14:47:38.869979 18870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:40023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:47:38.887352 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33987
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:38.967690 19066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:38.967968 19066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:38.968012 19066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:38.969187 19066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:38.969261 19066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:38.969333 19066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:47:38.970995 19066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:38.971261 19066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:38.971522 19066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:38.976521 19074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:38.976521 19071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:38.976580 19072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:38.976773 19066 server_base.cc:1034] running on GCE node
I20260928 14:47:38.976995 19066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:38.977255 19066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:38.978398 19066 hybrid_clock.cc:648] HybridClock initialized: now 1790606858978387 us; error 31 us; skew 500 ppm
Sep 28 14:47:38 dist-test-slave-j968 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:38.980806 19066 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:47:38.981264 19066 webserver.cc:460] Webserver started at http://127.18.96.130:35261/ using document root <none> and password file <none>
I20260928 14:47:38.981482 19066 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:38.981547 19066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:38.981704 19066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:38.982745 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "60a447837e55477fa61a30643f0cafe8"
format_stamp: "Formatted at 2026-09-28 14:47:38 on dist-test-slave-j968"
server_key: "ac701cac6929fc6dd9ece8c247612e26"
server_key_iv: "5ceba8fdef14ba02bea641cc842640c0"
server_key_version: "encryptionkey@0"
I20260928 14:47:38.983109 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "60a447837e55477fa61a30643f0cafe8"
format_stamp: "Formatted at 2026-09-28 14:47:38 on dist-test-slave-j968"
server_key: "ac701cac6929fc6dd9ece8c247612e26"
server_key_iv: "5ceba8fdef14ba02bea641cc842640c0"
server_key_version: "encryptionkey@0"
I20260928 14:47:38.984519 19066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:47:38.985239 19081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:38.985405 19066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:38.985507 19066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "60a447837e55477fa61a30643f0cafe8"
format_stamp: "Formatted at 2026-09-28 14:47:38 on dist-test-slave-j968"
server_key: "ac701cac6929fc6dd9ece8c247612e26"
server_key_iv: "5ceba8fdef14ba02bea641cc842640c0"
server_key_version: "encryptionkey@0"
I20260928 14:47:38.985598 19066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:39.003590 19066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:39.005539 19066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:39.005716 19066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:39.005985 19066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:39.006310 19066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:39.006364 19066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:39.006404 19066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:39.006433 19066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:39.012207 19066 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:45901
I20260928 14:47:39.012320 19194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:45901 every 8 connection(s)
I20260928 14:47:39.012627 19066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:39.021669 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:39.013350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:33039 (local address 127.18.96.190:33987)
0928 14:47:39.013437 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:39.013439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:39.013639 (+ 200us) server_negotiation.cc:407] Connection header received
0928 14:47:39.014100 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:39.014101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:39.014112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:39.014164 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:39.015155 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.015648 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.016773 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.016967 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.017024 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:39.019101 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:39.019102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:39.019103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:39.019108 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:39.020529 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:39.020901 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:39.020902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:39.020902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:39.020949 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:39.021165 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:39.021166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:39.021166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:39.021264 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:39.021313 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:39.021536 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 14:47:39.021604 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:47:39.021912 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:39.013310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:33987 (local address 127.18.96.130:33039)
0928 14:47:39.013502 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:39.013540 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:39.014007 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:39.014243 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:39.014246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:39.014824 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:39.015041 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:39.015046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.015729 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.015730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:39.016697 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:39.016699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.017042 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.017043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:39.017137 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:39.017667 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:39.017675 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:39.019005 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:39.020611 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:39.020617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:39.020618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:39.020826 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:39.021015 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:39.021015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:39.021016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:39.021111 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:39.021317 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:39.021321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:39.021427 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 14:47:39.021513 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:47:39.021627 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260928 14:47:39.022504 19195 heartbeater.cc:347] Connected to a master server at 127.18.96.190:33987
I20260928 14:47:39.022531 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 19066
I20260928 14:47:39.022607 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:47:39.022611 19195 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:39.022782 18818 external_mini_cluster.cc:1428] Setting key 865a36864303d647f3c6c2e86d4b040c
I20260928 14:47:39.022799 19195 heartbeater.cc:511] Master 127.18.96.190:33987 requested a full tablet report, sending...
I20260928 14:47:39.023272 18869 ts_manager.cc:194] Registered new tserver with Master: 60a447837e55477fa61a30643f0cafe8 (127.18.96.130:45901)
I20260928 14:47:39.023660 18869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:33039
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:47:39.047442 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33987
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:39.127281 19200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:39.127553 19200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:39.127593 19200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:39.128737 19200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:39.128809 19200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:39.128851 19200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:47:39.130295 19200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:39.130558 19200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:39.130793 19200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:39.136027 19206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:39.136101 19208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:39.136139 19200 server_base.cc:1034] running on GCE node
W20260928 14:47:39.136034 19205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:39.136458 19200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:39.136708 19200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:39.137842 19200 hybrid_clock.cc:648] HybridClock initialized: now 1790606859137830 us; error 28 us; skew 500 ppm
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606859, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:39.140386 19200 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:47:39.140820 19200 webserver.cc:460] Webserver started at http://127.18.96.131:38513/ using document root <none> and password file <none>
I20260928 14:47:39.141000 19200 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:39.141038 19200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:39.141203 19200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:39.142279 19200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "1dfd5cf001f14aaa81715ee79895b294"
format_stamp: "Formatted at 2026-09-28 14:47:39 on dist-test-slave-j968"
server_key: "87b35942977ec6ffb9ba415503aa0e31"
server_key_iv: "03761b7634ce410e3ab9732fe258abb1"
server_key_version: "encryptionkey@0"
I20260928 14:47:39.142633 19200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "1dfd5cf001f14aaa81715ee79895b294"
format_stamp: "Formatted at 2026-09-28 14:47:39 on dist-test-slave-j968"
server_key: "87b35942977ec6ffb9ba415503aa0e31"
server_key_iv: "03761b7634ce410e3ab9732fe258abb1"
server_key_version: "encryptionkey@0"
I20260928 14:47:39.143991 19200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:47:39.144719 19215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:39.144886 19200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:47:39.145011 19200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "1dfd5cf001f14aaa81715ee79895b294"
format_stamp: "Formatted at 2026-09-28 14:47:39 on dist-test-slave-j968"
server_key: "87b35942977ec6ffb9ba415503aa0e31"
server_key_iv: "03761b7634ce410e3ab9732fe258abb1"
server_key_version: "encryptionkey@0"
I20260928 14:47:39.145071 19200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:39.176146 19200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:39.178583 19200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:39.178700 19200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:39.178903 19200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:39.179242 19200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:39.179301 19200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:39.179345 19200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:39.179369 19200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:39.185561 19200 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:39353
I20260928 14:47:39.185606 19328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:39353 every 8 connection(s)
I20260928 14:47:39.185971 19200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:47:39.192294 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 19200
I20260928 14:47:39.192376 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606859, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:39.192551 18818 external_mini_cluster.cc:1428] Setting key ad997368bd54ecd593906b7f2980241b
I20260928 14:47:39.195996 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:39.186693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:48949 (local address 127.18.96.190:33987)
0928 14:47:39.186785 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:39.186787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:39.186981 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:47:39.187434 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:39.187435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:39.187445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:39.187525 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:39.188610 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.189117 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.190288 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.190494 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.190565 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:39.193031 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:39.193032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:39.193032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:39.193041 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:39.194534 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:39.195027 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:39.195027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:39.195028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:39.195087 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:39.195340 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:39.195341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:39.195341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:39.195453 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:39.195504 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:39.195770 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 14:47:39.195839 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:47:39.196275 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:39.186666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:33987 (local address 127.18.96.131:48949)
0928 14:47:39.186835 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:39.186879 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:39.187354 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:39.187584 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:39.187587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:39.188300 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:39.188508 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:39.188514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.189205 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.189206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:39.190162 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:39.190165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.190601 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.190602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:39.190725 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:39.191222 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:39.191230 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:39.192902 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:39.194639 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:39.194646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:39.194648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:39.194943 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:39.195168 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:39.195168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:39.195169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:39.195283 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:39.195497 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:39.195500 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:39.195654 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 14:47:39.195751 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:47:39.195962 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260928 14:47:39.196836 19329 heartbeater.cc:347] Connected to a master server at 127.18.96.190:33987
I20260928 14:47:39.196934 19329 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:39.197153 19329 heartbeater.cc:511] Master 127.18.96.190:33987 requested a full tablet report, sending...
I20260928 14:47:39.197505 18869 ts_manager.cc:194] Registered new tserver with Master: 1dfd5cf001f14aaa81715ee79895b294 (127.18.96.131:39353)
I20260928 14:47:39.197896 18869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:48949
I20260928 14:47:39.204694 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606859, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606859, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:47:39.228952 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:39.221302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.18.96.129:40297)
0928 14:47:39.221480 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:39.221485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:39.221495 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:47:39.221524 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:39.221526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:39.221548 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:39.221649 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:39.222252 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.223002 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.223752 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.224174 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.224266 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:39.226494 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:39.226501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:39.226501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:39.226508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:39.227996 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:39.228296 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:39.228298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:39.228299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:39.228351 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:39.228508 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:39.228509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:39.228509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:39.228644 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:39.228758 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:39.228811 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 14:47:39.228872 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
W20260928 14:47:39.229336 18975 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53430
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[18825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[18825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:47:39.252120 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:39.244609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.18.96.129:40297)
0928 14:47:39.244683 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:39.244684 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:39.244742 (+ 58us) server_negotiation.cc:407] Connection header received
0928 14:47:39.244794 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:39.244795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:39.244805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:39.244855 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:39.245390 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.245934 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.246694 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:39.246939 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:39.246994 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:39.249682 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:39.249683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:39.249683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:39.249689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:39.251012 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:39.251337 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:39.251338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:39.251338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:39.251386 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:39.251585 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:39.251586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:39.251586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:39.251671 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:39.251754 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:39.251828 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 14:47:39.251948 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:47:39.252722 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 18932
I20260928 14:47:39.258265 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 19066
I20260928 14:47:39.265179 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 19200
I20260928 14:47:39.269452 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 18838
2026-09-28T14:47:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1915 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[19343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[19343](info): set up 2 sockets
Sep 28 14:47:39 dist-test-slave-j968 krb5kdc[19343](info): commencing operation
krb5kdc: starting...
W20260928 14:47:40.308363 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:47:40 dist-test-slave-j968 krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:47:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:47:40.426352 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:41343
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:40.507387 19356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:40.507642 19356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:40.507679 19356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:40.508909 19356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:47:40.508978 19356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:40.509002 19356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:40.509024 19356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:47:40.509051 19356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:47:40.510519 19356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:41343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:40.510795 19356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:40.511027 19356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:40.516057 19362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:40.516042 19361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:40.516031 19364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:40.516116 19356 server_base.cc:1034] running on GCE node
I20260928 14:47:40.516501 19356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:40.516801 19356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:40.517974 19356 hybrid_clock.cc:648] HybridClock initialized: now 1790606860517966 us; error 37 us; skew 500 ppm
Sep 28 14:47:40 dist-test-slave-j968 krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:40.520690 19356 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:47:40.521178 19356 webserver.cc:460] Webserver started at http://127.18.96.190:41091/ using document root <none> and password file <none>
I20260928 14:47:40.521410 19356 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:40.521478 19356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:40.521632 19356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:40.522716 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "84226d27e88046bdbe3a5693c747ca7e"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-j968"
server_key: "e23a663c3b31b07ab00e1aa9afdc3888"
server_key_iv: "b252444181892c7efd14bf2f8944e275"
server_key_version: "encryptionkey@0"
I20260928 14:47:40.523082 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "84226d27e88046bdbe3a5693c747ca7e"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-j968"
server_key: "e23a663c3b31b07ab00e1aa9afdc3888"
server_key_iv: "b252444181892c7efd14bf2f8944e275"
server_key_version: "encryptionkey@0"
I20260928 14:47:40.524528 19356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:47:40.525282 19371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:40.525449 19356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:40.525548 19356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "84226d27e88046bdbe3a5693c747ca7e"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-j968"
server_key: "e23a663c3b31b07ab00e1aa9afdc3888"
server_key_iv: "b252444181892c7efd14bf2f8944e275"
server_key_version: "encryptionkey@0"
I20260928 14:47:40.525641 19356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:40.547430 19356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:40.549878 19356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:40.550055 19356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:40.554186 19356 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:41343
I20260928 14:47:40.554224 19423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:41343 every 8 connection(s)
I20260928 14:47:40.554636 19356 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:47:40.555629 19424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:40.558019 19424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e: Bootstrap starting.
I20260928 14:47:40.558637 19424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:40.558935 19424 log.cc:824] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:40.559679 19424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e: No bootstrap required, opened a new log
I20260928 14:47:40.561345 19424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84226d27e88046bdbe3a5693c747ca7e" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41343 } }
I20260928 14:47:40.561522 19424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:40.561573 19424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84226d27e88046bdbe3a5693c747ca7e, State: Initialized, Role: FOLLOWER
I20260928 14:47:40.561615 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 19356
I20260928 14:47:40.561702 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:47:40.561772 19424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84226d27e88046bdbe3a5693c747ca7e" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41343 } }
I20260928 14:47:40.561873 19424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:47:40.561896 18818 external_mini_cluster.cc:1428] Setting key c8104c16111b9a509a24308385f612a2
I20260928 14:47:40.561916 19424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:47:40.561985 19424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:40.563210 19424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84226d27e88046bdbe3a5693c747ca7e" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41343 } }
I20260928 14:47:40.563333 19424 leader_election.cc:302] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84226d27e88046bdbe3a5693c747ca7e; no voters:
I20260928 14:47:40.563510 19424 leader_election.cc:288] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:47:40.563539 19429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:40.563688 19429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [term 1 LEADER]: Becoming Leader. State: Replica: 84226d27e88046bdbe3a5693c747ca7e, State: Running, Role: LEADER
I20260928 14:47:40.563779 19424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:47:40.563782 19429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84226d27e88046bdbe3a5693c747ca7e" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41343 } }
I20260928 14:47:40.564392 19431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84226d27e88046bdbe3a5693c747ca7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84226d27e88046bdbe3a5693c747ca7e" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41343 } } }
I20260928 14:47:40.564405 19430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84226d27e88046bdbe3a5693c747ca7e. Latest consensus state: current_term: 1 leader_uuid: "84226d27e88046bdbe3a5693c747ca7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84226d27e88046bdbe3a5693c747ca7e" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41343 } } }
I20260928 14:47:40.564579 19430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:40.564651 19431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e [sys.catalog]: This master's current role is: LEADER
W20260928 14:47:40.565687 19444 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:47:40.565744 19444 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:47:40.566392 19438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:47:40.566599 19438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:47:40.568034 19438 catalog_manager.cc:1347] Generated new cluster ID: 7e350580619b4a20b346a2d80b57c6c0
I20260928 14:47:40.568091 19438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:47:40 dist-test-slave-j968 krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:40.573865 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:40.562438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.18.96.190:41343)
0928 14:47:40.562752 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:40.562764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:40.562804 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:47:40.562940 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:40.562948 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:40.563011 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:40.563170 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:40.564040 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.565582 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.566390 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.566996 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.567114 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:40.570035 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:40.570040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:40.570041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:40.570059 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:40.572188 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:40.572579 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:40.572583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:40.572583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:40.572708 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:40.572909 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:40.572909 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:40.572910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:40.573250 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:40.573413 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:40.573515 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:47:40.573663 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":167,"threads_started":1}
I20260928 14:47:40.582405 19438 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:47:40.583119 19438 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:47:40.585891 19438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84226d27e88046bdbe3a5693c747ca7e: Generated new TSK 0
I20260928 14:47:40.586072 19438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:47:40.618438 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41343
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:40.699417 19450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:40.699625 19450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:40.699651 19450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:40.700850 19450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:40.700889 19450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:40.700917 19450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:47:40.702431 19450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:40.702677 19450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:40.702919 19450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:40.707952 19455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:40.707993 19458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:40.707998 19456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:40.708329 19450 server_base.cc:1034] running on GCE node
I20260928 14:47:40.708580 19450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:40.708835 19450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:40.710024 19450 hybrid_clock.cc:648] HybridClock initialized: now 1790606860710008 us; error 24 us; skew 500 ppm
Sep 28 14:47:40 dist-test-slave-j968 krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:40.712738 19450 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:47:40.713228 19450 webserver.cc:460] Webserver started at http://127.18.96.129:35047/ using document root <none> and password file <none>
I20260928 14:47:40.713452 19450 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:40.713521 19450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:40.713670 19450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:40.714721 19450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "048d3a5baa884ca3afb9e185a68f47f3"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-j968"
server_key: "a9e815d5d8ba10a06140c9a5f84c5ac6"
server_key_iv: "e510eafd29ff6753150e030f08b3bd21"
server_key_version: "encryptionkey@0"
I20260928 14:47:40.715080 19450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "048d3a5baa884ca3afb9e185a68f47f3"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-j968"
server_key: "a9e815d5d8ba10a06140c9a5f84c5ac6"
server_key_iv: "e510eafd29ff6753150e030f08b3bd21"
server_key_version: "encryptionkey@0"
I20260928 14:47:40.716462 19450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:47:40.717190 19465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:40.717355 19450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:40.717458 19450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "048d3a5baa884ca3afb9e185a68f47f3"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-j968"
server_key: "a9e815d5d8ba10a06140c9a5f84c5ac6"
server_key_iv: "e510eafd29ff6753150e030f08b3bd21"
server_key_version: "encryptionkey@0"
I20260928 14:47:40.717528 19450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:40.739605 19450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:40.742172 19450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:40.742341 19450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:40.742594 19450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:40.742910 19450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:40.742964 19450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:40.743006 19450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:40.743033 19450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:40.749074 19450 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:46625
I20260928 14:47:40.749116 19578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:46625 every 8 connection(s)
I20260928 14:47:40.749462 19450 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:47:40.753727 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 19450
I20260928 14:47:40.753806 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:47:40.754001 18818 external_mini_cluster.cc:1428] Setting key 83c23ffff2903a8a4b6ae38fd26670ec
Sep 28 14:47:40 dist-test-slave-j968 krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:40.760142 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:40.750286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:37051 (local address 127.18.96.190:41343)
0928 14:47:40.750333 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:40.750335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:40.750520 (+ 185us) server_negotiation.cc:407] Connection header received
0928 14:47:40.751426 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:40.751428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:40.751442 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:40.751515 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:40.752144 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.752654 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.753932 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.754280 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.754347 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:40.756892 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:40.756894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:40.756894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:40.756903 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:40.758548 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:40.758992 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:40.758993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:40.758994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:40.759061 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:40.759303 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:40.759304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:40.759304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:40.759458 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:40.759538 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:40.759783 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 14:47:40.759903 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:47:40.760275 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:40.750230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:41343 (local address 127.18.96.129:37051)
0928 14:47:40.750389 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:40.750432 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:40.751329 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:40.751581 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:40.751585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:40.751840 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:40.752058 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:40.752062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.752733 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.752734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:40.753839 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:40.753844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.754353 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.754354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:40.754520 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:40.755142 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:40.755151 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:40.756787 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:40.758654 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:40.758662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:40.758664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:40.758904 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:40.759131 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:40.759132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:40.759132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:40.759234 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:40.759528 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:40.759533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:40.759648 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 14:47:40.759731 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:47:40.759851 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260928 14:47:40.760906 19579 heartbeater.cc:347] Connected to a master server at 127.18.96.190:41343
I20260928 14:47:40.761027 19579 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:40.761216 19579 heartbeater.cc:511] Master 127.18.96.190:41343 requested a full tablet report, sending...
I20260928 14:47:40.761806 19388 ts_manager.cc:194] Registered new tserver with Master: 048d3a5baa884ca3afb9e185a68f47f3 (127.18.96.129:46625)
I20260928 14:47:40.762530 19388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:37051
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:47:40.782625 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41343
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:40.862413 19584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:40.862618 19584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:40.862643 19584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:40.863730 19584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:40.863781 19584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:40.863808 19584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:47:40.865403 19584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:40.865618 19584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:40.865826 19584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:40.870733 19589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:40.870740 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:40.870896 19584 server_base.cc:1034] running on GCE node
W20260928 14:47:40.870790 19590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:40.871248 19584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:40.871490 19584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:40.872653 19584 hybrid_clock.cc:648] HybridClock initialized: now 1790606860872649 us; error 34 us; skew 500 ppm
Sep 28 14:47:40 dist-test-slave-j968 krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:40.875146 19584 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:47:40.875617 19584 webserver.cc:460] Webserver started at http://127.18.96.130:38953/ using document root <none> and password file <none>
I20260928 14:47:40.875851 19584 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:40.876026 19584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:40.876165 19584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:40.877195 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "754be11cefa04a8093b1fd9f544e0089"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-j968"
server_key: "8ade6c600df54e094c8c9c3d705e45db"
server_key_iv: "1f5c2d25390c87b1552658bc91ac0689"
server_key_version: "encryptionkey@0"
I20260928 14:47:40.877556 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "754be11cefa04a8093b1fd9f544e0089"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-j968"
server_key: "8ade6c600df54e094c8c9c3d705e45db"
server_key_iv: "1f5c2d25390c87b1552658bc91ac0689"
server_key_version: "encryptionkey@0"
I20260928 14:47:40.878824 19584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:47:40.879570 19599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:40.879743 19584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:40.879832 19584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "754be11cefa04a8093b1fd9f544e0089"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-j968"
server_key: "8ade6c600df54e094c8c9c3d705e45db"
server_key_iv: "1f5c2d25390c87b1552658bc91ac0689"
server_key_version: "encryptionkey@0"
I20260928 14:47:40.879997 19584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:40.895692 19584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:40.897652 19584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:40.897814 19584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:40.898043 19584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:40.898344 19584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:40.898402 19584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:40.898443 19584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:40.898470 19584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:40.904290 19584 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:36541
I20260928 14:47:40.904388 19712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:36541 every 8 connection(s)
I20260928 14:47:40.904670 19584 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:47:40.907380 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 19584
I20260928 14:47:40.907491 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:47:40.907665 18818 external_mini_cluster.cc:1428] Setting key a0f4464a27df642366a6b6175a746ff1
Sep 28 14:47:40 dist-test-slave-j968 krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:40.915362 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:40.905426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:59561 (local address 127.18.96.190:41343)
0928 14:47:40.905514 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:40.905516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:40.905709 (+ 193us) server_negotiation.cc:407] Connection header received
0928 14:47:40.906706 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:40.906707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:40.906717 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:40.906776 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:40.907417 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.908153 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.909665 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.910007 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.910097 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:40.912250 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:40.912252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:40.912252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:40.912260 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:40.913867 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:40.914358 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:40.914359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:40.914360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:40.914410 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:40.914699 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:40.914700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:40.914700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:40.914816 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:40.914868 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:40.915211 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 14:47:40.915298 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:47:40.915633 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:40.905408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:41343 (local address 127.18.96.130:59561)
0928 14:47:40.905591 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:40.905626 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:40.906523 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:40.906847 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:40.906850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:40.907103 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:40.907309 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:40.907313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.908240 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.908241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:40.909580 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:40.909583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.910106 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.910106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:40.910209 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:40.910750 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:40.910768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:40.912155 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:40.913950 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:40.913958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:40.913960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:40.914276 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:40.914483 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:40.914484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:40.914484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:40.914621 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:40.914886 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:40.914893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:40.915067 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 14:47:40.915196 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 14:47:40.915369 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260928 14:47:40.916206 19713 heartbeater.cc:347] Connected to a master server at 127.18.96.190:41343
I20260928 14:47:40.916308 19713 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:40.916508 19713 heartbeater.cc:511] Master 127.18.96.190:41343 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
I20260928 14:47:40.916890 19388 ts_manager.cc:194] Registered new tserver with Master: 754be11cefa04a8093b1fd9f544e0089 (127.18.96.130:36541)
I20260928 14:47:40.917264 19388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:59561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:47:40.934794 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41343
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:41.015070 19718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:41.015301 19718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:41.015326 19718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:41.016572 19718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:41.016620 19718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:41.016647 19718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:47:41.018181 19718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:41.018441 19718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:41.018676 19718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:41.024194 19724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:41.024196 19723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:41.024240 19726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:41.024638 19718 server_base.cc:1034] running on GCE node
I20260928 14:47:41.024923 19718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:41.025285 19718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:41.026448 19718 hybrid_clock.cc:648] HybridClock initialized: now 1790606861026437 us; error 31 us; skew 500 ppm
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:41.037524 19718 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:47:41.037993 19718 webserver.cc:460] Webserver started at http://127.18.96.131:44195/ using document root <none> and password file <none>
I20260928 14:47:41.038265 19718 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:41.038338 19718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:41.038501 19718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:41.039623 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "548abe70a12a4adfb4b67194c8e0077d"
format_stamp: "Formatted at 2026-09-28 14:47:41 on dist-test-slave-j968"
server_key: "448896ca0d6d098d6fbf8d550becd2ff"
server_key_iv: "58a1a1d7747157732ada90a674a31fed"
server_key_version: "encryptionkey@0"
I20260928 14:47:41.040030 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "548abe70a12a4adfb4b67194c8e0077d"
format_stamp: "Formatted at 2026-09-28 14:47:41 on dist-test-slave-j968"
server_key: "448896ca0d6d098d6fbf8d550becd2ff"
server_key_iv: "58a1a1d7747157732ada90a674a31fed"
server_key_version: "encryptionkey@0"
I20260928 14:47:41.041535 19718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:41.042488 19733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:41.042716 19718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:41.042838 19718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "548abe70a12a4adfb4b67194c8e0077d"
format_stamp: "Formatted at 2026-09-28 14:47:41 on dist-test-slave-j968"
server_key: "448896ca0d6d098d6fbf8d550becd2ff"
server_key_iv: "58a1a1d7747157732ada90a674a31fed"
server_key_version: "encryptionkey@0"
I20260928 14:47:41.042917 19718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:41.066262 19718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:41.068593 19718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:41.068753 19718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:41.068988 19718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:41.069319 19718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:41.069375 19718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:41.069416 19718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:41.069511 19718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:41.075184 19718 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:38549
I20260928 14:47:41.075255 19846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:38549 every 8 connection(s)
I20260928 14:47:41.075595 19718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:47:41.079746 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 19718
I20260928 14:47:41.079840 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:47:41.080024 18818 external_mini_cluster.cc:1428] Setting key 6ea2bce0274723a74595a77f21c6f8d5
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:41.085057 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.076340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:35783 (local address 127.18.96.190:41343)
0928 14:47:41.076418 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.076421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.076610 (+ 189us) server_negotiation.cc:407] Connection header received
0928 14:47:41.077504 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.077505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.077515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.077582 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:41.078140 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.078628 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.079613 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.079927 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.080001 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.082243 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:41.082244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:41.082244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:41.082251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:41.083639 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.084119 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.084121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.084121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.084182 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.084427 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.084428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.084429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.084544 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:41.084619 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.084886 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.084968 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:47:41.085270 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.076296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:41343 (local address 127.18.96.131:35783)
0928 14:47:41.076481 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:41.076517 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:41.077411 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:41.077617 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:41.077621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:41.077877 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:41.078069 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.078074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.078703 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.078704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.079506 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.079508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.080026 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.080027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.080189 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.080795 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:41.080805 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:41.082165 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:41.083725 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:41.083731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:41.083733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:41.084026 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:41.084247 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:41.084248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:41.084249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:41.084360 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:41.084611 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:41.084617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:41.084757 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 14:47:41.084863 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:47:41.085017 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260928 14:47:41.085677 19847 heartbeater.cc:347] Connected to a master server at 127.18.96.190:41343
I20260928 14:47:41.085762 19847 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:41.085947 19847 heartbeater.cc:511] Master 127.18.96.190:41343 requested a full tablet report, sending...
I20260928 14:47:41.086316 19388 ts_manager.cc:194] Registered new tserver with Master: 548abe70a12a4adfb4b67194c8e0077d (127.18.96.131:38549)
I20260928 14:47:41.086712 19388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:35783
I20260928 14:47:41.091636 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:47:41.102275 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.093197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.18.96.190:41343)
0928 14:47:41.093255 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.093257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.093408 (+ 151us) server_negotiation.cc:407] Connection header received
0928 14:47:41.093522 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.093522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.093559 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.093646 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:41.094320 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.095122 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.096379 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.096698 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.096792 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.098561 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:41.098563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:41.098563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:41.098572 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:41.100644 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.101309 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.101310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.101311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.101391 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.101615 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.101616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.101617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.101828 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:41.101963 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.102069 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.102155 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:47:41.105511 19388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:47:41.108412 19388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:41.115320 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.111000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:38549 (local address 127.0.0.1:44746)
0928 14:47:41.111129 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:41.111140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:41.111191 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:41.111835 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:41.111839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:41.111841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:41.111950 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.111953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.112859 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.112860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.114178 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.114180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.114990 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.114991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.115073 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.115086 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:47:41.115192 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:47:41.115247 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260928 14:47:41.115329 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.111386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.18.96.131:38549)
0928 14:47:41.111565 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.111570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.111580 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:47:41.111623 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.111626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.111643 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.111725 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:41.112042 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.112740 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.114265 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.114899 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.114990 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.115054 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.115209 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.115250 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260928 14:47:41.115545 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.110906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:36541 (local address 127.0.0.1:33408)
0928 14:47:41.111021 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:41.111081 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:41.111149 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:41.112013 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:41.112014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:41.112016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:41.112048 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.112049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.113354 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.113355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.114393 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.114396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.115322 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.115324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.115387 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.115391 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:47:41.115424 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 14:47:41.115467 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
I20260928 14:47:41.115772 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.111486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.18.96.129:46625)
0928 14:47:41.111644 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.111649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.111661 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:47:41.111702 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.111704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.111720 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.112055 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:41.112140 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.113333 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.114586 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.115263 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.115511 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.115594 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.115622 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.115672 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260928 14:47:41.116132 19781 tablet_service.cc:1467] Processing CreateTablet for tablet 39b030103cd54d6ba1eedabafa95e7bc (DEFAULT_TABLE table=test-table [id=cf31825e3d1e4a158a4116c9691d6d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:41.116282 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.111176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:46625 (local address 127.0.0.1:34306)
0928 14:47:41.111288 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:41.111298 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:41.111340 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:41.111846 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:41.111847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:41.111850 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:41.111967 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.111969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.113432 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.113433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.114498 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.114500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.115349 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.115350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.115411 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.115415 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:47:41.115443 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 14:47:41.115472 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":73,"threads_started":1}
I20260928 14:47:41.116544 19781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b030103cd54d6ba1eedabafa95e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:41.116925 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.111064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.18.96.130:36541)
0928 14:47:41.111751 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.111755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.111763 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:47:41.111802 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.111805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.111818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.112132 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:41.112216 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.113238 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.114502 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.115217 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.116652 (+ 1435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.116737 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.116767 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.116819 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":92,"threads_started":1}
I20260928 14:47:41.117013 19513 tablet_service.cc:1467] Processing CreateTablet for tablet 39b030103cd54d6ba1eedabafa95e7bc (DEFAULT_TABLE table=test-table [id=cf31825e3d1e4a158a4116c9691d6d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:41.117332 19513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b030103cd54d6ba1eedabafa95e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:41.117554 19647 tablet_service.cc:1467] Processing CreateTablet for tablet 39b030103cd54d6ba1eedabafa95e7bc (DEFAULT_TABLE table=test-table [id=cf31825e3d1e4a158a4116c9691d6d2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:41.117872 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b030103cd54d6ba1eedabafa95e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:41.121248 19865 tablet_bootstrap.cc:492] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3: Bootstrap starting.
I20260928 14:47:41.121619 19866 tablet_bootstrap.cc:492] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089: Bootstrap starting.
I20260928 14:47:41.121827 19864 tablet_bootstrap.cc:492] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d: Bootstrap starting.
I20260928 14:47:41.122006 19865 tablet_bootstrap.cc:654] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:41.122529 19864 tablet_bootstrap.cc:654] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:41.122583 19865 log.cc:824] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:41.122558 19866 tablet_bootstrap.cc:654] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:41.123134 19864 log.cc:824] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:41.123346 19866 log.cc:824] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:41.123445 19865 tablet_bootstrap.cc:492] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3: No bootstrap required, opened a new log
I20260928 14:47:41.123505 19865 ts_tablet_manager.cc:1397] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:47:41.124130 19864 tablet_bootstrap.cc:492] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d: No bootstrap required, opened a new log
I20260928 14:47:41.124204 19864 ts_tablet_manager.cc:1397] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:47:41.124408 19866 tablet_bootstrap.cc:492] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089: No bootstrap required, opened a new log
I20260928 14:47:41.124526 19866 ts_tablet_manager.cc:1397] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:47:41.126281 19864 raft_consensus.cc:348] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } }
I20260928 14:47:41.126202 19865 raft_consensus.cc:348] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } }
I20260928 14:47:41.126470 19864 raft_consensus.cc:374] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:41.126470 19865 raft_consensus.cc:374] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:41.126523 19864 raft_consensus.cc:729] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548abe70a12a4adfb4b67194c8e0077d, State: Initialized, Role: FOLLOWER
I20260928 14:47:41.126546 19865 raft_consensus.cc:729] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048d3a5baa884ca3afb9e185a68f47f3, State: Initialized, Role: FOLLOWER
I20260928 14:47:41.126645 19864 consensus_queue.cc:260] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } }
I20260928 14:47:41.126729 19865 consensus_queue.cc:260] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } }
I20260928 14:47:41.126931 19864 ts_tablet_manager.cc:1428] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:47:41.126940 19847 heartbeater.cc:503] Master 127.18.96.190:41343 was elected leader, sending a full tablet report...
I20260928 14:47:41.127096 19579 heartbeater.cc:503] Master 127.18.96.190:41343 was elected leader, sending a full tablet report...
I20260928 14:47:41.127328 19865 ts_tablet_manager.cc:1428] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 14:47:41.126832 19866 raft_consensus.cc:348] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } }
I20260928 14:47:41.128391 19866 raft_consensus.cc:374] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:41.128491 19866 raft_consensus.cc:729] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754be11cefa04a8093b1fd9f544e0089, State: Initialized, Role: FOLLOWER
I20260928 14:47:41.128609 19866 consensus_queue.cc:260] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } }
I20260928 14:47:41.128862 19866 ts_tablet_manager.cc:1428] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:47:41.129032 19713 heartbeater.cc:503] Master 127.18.96.190:41343 was elected leader, sending a full tablet report...
W20260928 14:47:41.155292 19714 tablet.cc:2369] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:47:41.250167 19580 tablet.cc:2369] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:47:41.275354 19871 raft_consensus.cc:482] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:41.275480 19871 raft_consensus.cc:504] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } }
I20260928 14:47:41.275795 19871 leader_election.cc:288] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 754be11cefa04a8093b1fd9f544e0089 (127.18.96.130:36541), 548abe70a12a4adfb4b67194c8e0077d (127.18.96.131:38549)
I20260928 14:47:41.278852 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.276028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:38549 (local address 127.18.96.129:38361)
0928 14:47:41.276167 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:41.276179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:41.276252 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:41.276431 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:41.276432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:41.276434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:41.276518 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.276520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.277162 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.277163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.278016 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.278018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.278587 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.278587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.278661 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.278664 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:47:41.278728 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:47:41.278767 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":37,"thread_start_us":64,"threads_started":1}
I20260928 14:47:41.278838 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.276166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:38361 (local address 127.18.96.131:38549)
0928 14:47:41.276219 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.276221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.276233 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:47:41.276327 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.276328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.276336 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.276421 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:41.276619 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.277079 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.278113 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.278481 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.278579 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.278588 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.278733 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.278765 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:47:41.279084 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.276012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:36541 (local address 127.18.96.129:38663)
0928 14:47:41.276147 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:41.276160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:41.276256 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:41.276427 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:41.276428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:41.276430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:41.276526 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.276527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.277172 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.277173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.278018 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.278020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.278730 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.278731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.278817 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.278821 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:47:41.278942 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 14:47:41.278987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":12,"thread_start_us":64,"threads_started":1}
I20260928 14:47:41.279070 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.276096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:38663 (local address 127.18.96.130:36541)
0928 14:47:41.276153 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.276156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.276230 (+ 74us) server_negotiation.cc:407] Connection header received
0928 14:47:41.276339 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.276340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.276347 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.276408 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:41.276598 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.277086 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.278113 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.278493 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.278722 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.278733 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.278963 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.279004 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:47:41.279501 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b030103cd54d6ba1eedabafa95e7bc" candidate_uuid: "048d3a5baa884ca3afb9e185a68f47f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754be11cefa04a8093b1fd9f544e0089" is_pre_election: true
I20260928 14:47:41.279518 19801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b030103cd54d6ba1eedabafa95e7bc" candidate_uuid: "048d3a5baa884ca3afb9e185a68f47f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548abe70a12a4adfb4b67194c8e0077d" is_pre_election: true
I20260928 14:47:41.279661 19667 raft_consensus.cc:2454] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 048d3a5baa884ca3afb9e185a68f47f3 in term 0.
I20260928 14:47:41.279680 19801 raft_consensus.cc:2454] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 048d3a5baa884ca3afb9e185a68f47f3 in term 0.
I20260928 14:47:41.279927 19468 leader_election.cc:302] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048d3a5baa884ca3afb9e185a68f47f3, 548abe70a12a4adfb4b67194c8e0077d; no voters:
I20260928 14:47:41.280078 19871 raft_consensus.cc:2781] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:41.280138 19871 raft_consensus.cc:482] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:41.280165 19871 raft_consensus.cc:3037] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:41.280824 19871 raft_consensus.cc:504] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } }
I20260928 14:47:41.280969 19871 leader_election.cc:288] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [CANDIDATE]: Term 1 election: Requested vote from peers 754be11cefa04a8093b1fd9f544e0089 (127.18.96.130:36541), 548abe70a12a4adfb4b67194c8e0077d (127.18.96.131:38549)
I20260928 14:47:41.281109 19801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b030103cd54d6ba1eedabafa95e7bc" candidate_uuid: "048d3a5baa884ca3afb9e185a68f47f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548abe70a12a4adfb4b67194c8e0077d"
I20260928 14:47:41.281143 19667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b030103cd54d6ba1eedabafa95e7bc" candidate_uuid: "048d3a5baa884ca3afb9e185a68f47f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "754be11cefa04a8093b1fd9f544e0089"
I20260928 14:47:41.281205 19801 raft_consensus.cc:3037] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:41.281239 19667 raft_consensus.cc:3037] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:41.281955 19801 raft_consensus.cc:2454] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 048d3a5baa884ca3afb9e185a68f47f3 in term 1.
I20260928 14:47:41.282090 19468 leader_election.cc:302] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048d3a5baa884ca3afb9e185a68f47f3, 548abe70a12a4adfb4b67194c8e0077d; no voters:
I20260928 14:47:41.282119 19667 raft_consensus.cc:2454] T 39b030103cd54d6ba1eedabafa95e7bc P 754be11cefa04a8093b1fd9f544e0089 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 048d3a5baa884ca3afb9e185a68f47f3 in term 1.
I20260928 14:47:41.282233 19871 raft_consensus.cc:2781] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:41.282418 19871 raft_consensus.cc:686] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [term 1 LEADER]: Becoming Leader. State: Replica: 048d3a5baa884ca3afb9e185a68f47f3, State: Running, Role: LEADER
I20260928 14:47:41.282526 19871 consensus_queue.cc:237] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } }
I20260928 14:47:41.283295 19388 catalog_manager.cc:5495] T 39b030103cd54d6ba1eedabafa95e7bc P 048d3a5baa884ca3afb9e185a68f47f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 048d3a5baa884ca3afb9e185a68f47f3 (127.18.96.129). New cstate: current_term: 1 leader_uuid: "048d3a5baa884ca3afb9e185a68f47f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "754be11cefa04a8093b1fd9f544e0089" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 36541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "548abe70a12a4adfb4b67194c8e0077d" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "048d3a5baa884ca3afb9e185a68f47f3" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46625 } health_report { overall_health: HEALTHY } } }
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:47:41.300203 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.293081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.18.96.129:46625)
0928 14:47:41.293143 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.293145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.293230 (+ 85us) server_negotiation.cc:407] Connection header received
0928 14:47:41.293330 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.293330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.293347 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.293393 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:41.293897 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.294346 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.295049 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.295221 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.295267 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.297563 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:41.297577 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:41.297577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:41.297583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:41.299205 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.299501 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.299503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.299504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.299578 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.299719 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.299719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.299719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.299837 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:41.300026 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.300077 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.300140 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:47:41.323192 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.315763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.18.96.129:46625)
0928 14:47:41.315829 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.315831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.315924 (+ 93us) server_negotiation.cc:407] Connection header received
0928 14:47:41.316017 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.316018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.316027 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.316119 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:41.316679 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.317154 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.317869 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.318061 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.318149 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.320439 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:41.320440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:41.320440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:41.320448 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:41.321977 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.322328 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.322329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.322330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.322388 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.322600 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.322601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.322602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.322706 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:41.322827 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.322999 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.323087 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:47:41.326244 19848 tablet.cc:2369] T 39b030103cd54d6ba1eedabafa95e7bc P 548abe70a12a4adfb4b67194c8e0077d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:47:41.345429 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.338198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.18.96.129:46625)
0928 14:47:41.338280 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.338282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.338389 (+ 107us) server_negotiation.cc:407] Connection header received
0928 14:47:41.338455 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.338456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.338467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.338562 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:41.339124 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.339602 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.340412 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.340612 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.340697 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.342916 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:41.342917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:41.342918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:41.342925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:41.344233 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.344610 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.344611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.344611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.344672 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.344885 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.344886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.344886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.344996 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:41.345127 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.345241 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.345323 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 14:47:41.345893 19493 tablet_service.cc:2908] Rejecting scan request for tablet 39b030103cd54d6ba1eedabafa95e7bc: Uninitialized: clean time has not yet been initialized
I20260928 14:47:41.346783 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 19450
I20260928 14:47:41.351716 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 19584
I20260928 14:47:41.356509 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 19718
I20260928 14:47:41.360637 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 19356
2026-09-28T14:47:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2092 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19889](info): set up 2 sockets
Sep 28 14:47:41 dist-test-slave-j968 krb5kdc[19889](info): commencing operation
krb5kdc: starting...
W20260928 14:47:42.400477 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:47:42 dist-test-slave-j968 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:47:42.515327 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:46547
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:42.599231 19902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:42.599476 19902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:42.599512 19902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:42.600677 19902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:47:42.600746 19902 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:47:42.600768 19902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:42.600790 19902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:42.600819 19902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:47:42.600842 19902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:47:42.602604 19902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:46547
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:46547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:42.602890 19902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:42.603111 19902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:42.607900 19908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:42.607981 19910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:42.607944 19907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:42.608022 19902 server_base.cc:1034] running on GCE node
I20260928 14:47:42.608268 19902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:42.608536 19902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:42.609696 19902 hybrid_clock.cc:648] HybridClock initialized: now 1790606862609682 us; error 28 us; skew 500 ppm
Sep 28 14:47:42 dist-test-slave-j968 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:42.615442 19902 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:47:42.615906 19902 webserver.cc:460] Webserver started at http://127.18.96.190:35551/ using document root <none> and password file <none>
I20260928 14:47:42.616145 19902 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:42.616211 19902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:42.616364 19902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:42.617394 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "b946d2ee718141879c79571900f56ddd"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-j968"
server_key: "77e26d8b107347308f63bb3b5a1cc0d8"
server_key_iv: "4aadee3a69ac5341d0089bd58406ea81"
server_key_version: "encryptionkey@0"
I20260928 14:47:42.617754 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "b946d2ee718141879c79571900f56ddd"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-j968"
server_key: "77e26d8b107347308f63bb3b5a1cc0d8"
server_key_iv: "4aadee3a69ac5341d0089bd58406ea81"
server_key_version: "encryptionkey@0"
I20260928 14:47:42.619030 19902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:47:42.619751 19917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.619946 19902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:42.620042 19902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "b946d2ee718141879c79571900f56ddd"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-j968"
server_key: "77e26d8b107347308f63bb3b5a1cc0d8"
server_key_iv: "4aadee3a69ac5341d0089bd58406ea81"
server_key_version: "encryptionkey@0"
I20260928 14:47:42.620136 19902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:42.627915 19902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:42.629578 19902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:42.629729 19902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:42.634145 19902 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:46547
I20260928 14:47:42.634239 19979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:46547 every 8 connection(s)
I20260928 14:47:42.634524 19902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:47:42.635155 19980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:42.637406 19980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd: Bootstrap starting.
I20260928 14:47:42.638005 19980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:42.638286 19980 log.cc:824] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:42.638935 19980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd: No bootstrap required, opened a new log
I20260928 14:47:42.639698 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 19902
I20260928 14:47:42.639788 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:47:42.639982 18818 external_mini_cluster.cc:1428] Setting key 5dc847a13a596d1aa54991117036eaf2
I20260928 14:47:42.641656 19980 raft_consensus.cc:348] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b946d2ee718141879c79571900f56ddd" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46547 } }
I20260928 14:47:42.641835 19980 raft_consensus.cc:374] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:42.641906 19980 raft_consensus.cc:729] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b946d2ee718141879c79571900f56ddd, State: Initialized, Role: FOLLOWER
I20260928 14:47:42.642086 19980 consensus_queue.cc:260] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b946d2ee718141879c79571900f56ddd" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46547 } }
I20260928 14:47:42.642180 19980 raft_consensus.cc:388] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:47:42.642217 19980 raft_consensus.cc:482] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:47:42.642268 19980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:42.642848 19980 raft_consensus.cc:504] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b946d2ee718141879c79571900f56ddd" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46547 } }
I20260928 14:47:42.642930 19980 leader_election.cc:302] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b946d2ee718141879c79571900f56ddd; no voters:
I20260928 14:47:42.643160 19980 leader_election.cc:288] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:47:42.643191 19985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:42.643296 19985 raft_consensus.cc:686] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [term 1 LEADER]: Becoming Leader. State: Replica: b946d2ee718141879c79571900f56ddd, State: Running, Role: LEADER
I20260928 14:47:42.643375 19985 consensus_queue.cc:237] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b946d2ee718141879c79571900f56ddd" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46547 } }
I20260928 14:47:42.643410 19980 sys_catalog.cc:564] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:47:42.643716 19985 sys_catalog.cc:455] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b946d2ee718141879c79571900f56ddd. Latest consensus state: current_term: 1 leader_uuid: "b946d2ee718141879c79571900f56ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b946d2ee718141879c79571900f56ddd" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46547 } } }
I20260928 14:47:42.643724 19987 sys_catalog.cc:455] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b946d2ee718141879c79571900f56ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b946d2ee718141879c79571900f56ddd" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46547 } } }
I20260928 14:47:42.643802 19985 sys_catalog.cc:458] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:42.643826 19987 sys_catalog.cc:458] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:42.644064 19991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:47:42.644574 19991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 14:47:42 dist-test-slave-j968 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:42.648844 19991 catalog_manager.cc:1347] Generated new cluster ID: 79ad22751d8f4427b178c54869f6515a
I20260928 14:47:42.648890 19991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:47:42.651595 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.640452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.18.96.190:46547)
0928 14:47:42.640605 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.640616 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.640646 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:47:42.641597 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.641604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.641652 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.641764 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:42.642529 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.643691 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.644792 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.645245 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.645362 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.647841 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:42.647844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:42.647844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:42.647854 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:42.649905 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.650365 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.650367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.650368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.650462 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.650672 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.650673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.650673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.650934 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:42.651072 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.651190 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.651405 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260928 14:47:42.661140 19991 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:47:42.661608 19991 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:47:42.665135 19991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd: Generated new TSK 0
I20260928 14:47:42.665303 19991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:47:42.691411 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:46547
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:42.772285 20006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:42.772527 20006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:42.772565 20006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:42.773728 20006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:42.773793 20006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:42.773826 20006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:47:42.773869 20006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:47:42.775272 20006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.190:46547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:42.775543 20006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:42.775780 20006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:42.780838 20014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:42.780865 20011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:42.780875 20012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:42.781128 20006 server_base.cc:1034] running on GCE node
I20260928 14:47:42.781358 20006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:42.781605 20006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:42.782778 20006 hybrid_clock.cc:648] HybridClock initialized: now 1790606862782766 us; error 27 us; skew 500 ppm
Sep 28 14:47:42 dist-test-slave-j968 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:42.785434 20006 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:47:42.785953 20006 webserver.cc:460] Webserver started at http://127.18.96.129:41225/ using document root <none> and password file <none>
I20260928 14:47:42.786183 20006 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:42.786249 20006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:42.786406 20006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:42.787524 20006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-j968"
server_key: "44df34ebd230f8b343f682582504944a"
server_key_iv: "1abfffa0d84c7fdb8ac25fa8b1539d11"
server_key_version: "encryptionkey@0"
I20260928 14:47:42.787957 20006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-j968"
server_key: "44df34ebd230f8b343f682582504944a"
server_key_iv: "1abfffa0d84c7fdb8ac25fa8b1539d11"
server_key_version: "encryptionkey@0"
I20260928 14:47:42.789335 20006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:42.790130 20021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.790313 20006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.790414 20006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-j968"
server_key: "44df34ebd230f8b343f682582504944a"
server_key_iv: "1abfffa0d84c7fdb8ac25fa8b1539d11"
server_key_version: "encryptionkey@0"
I20260928 14:47:42.790525 20006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:42.798473 20006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:42.800735 20006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:42.800894 20006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:42.801378 20006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:42.801443 20006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.801514 20006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:42.801544 20006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:47:42 dist-test-slave-j968 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:42.808297 20006 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:45909
I20260928 14:47:42.808400 20136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:45909 every 8 connection(s)
I20260928 14:47:42.808697 20006 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:47:42.811245 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.801827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:45441 (local address 127.18.96.190:46547)
0928 14:47:42.801908 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.801910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.802123 (+ 213us) server_negotiation.cc:407] Connection header received
0928 14:47:42.802588 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.802589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.802603 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.802693 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:42.803528 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.804407 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.805939 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.806318 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.806405 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.808719 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:42.808721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:42.808721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:42.808730 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:42.810148 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.810494 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.810495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.810495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.810543 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.810751 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.810751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.810752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.810849 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:42.810896 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.811117 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.811181 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:47:42.811384 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.801812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46547 (local address 127.18.96.129:45441)
0928 14:47:42.801984 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.802023 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.802497 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.802769 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.802773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.803140 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:42.803421 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.803427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.804495 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.804496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.805844 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.805848 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.806407 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.806408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.806526 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.807042 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:42.807050 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:42.808625 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:42.810219 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.810224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.810225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.810429 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.810607 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.810608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.810608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.810699 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.810901 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:42.810905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:42.811009 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.811090 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.811204 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":110,"threads_started":1}
I20260928 14:47:42.811925 20137 heartbeater.cc:347] Connected to a master server at 127.18.96.190:46547
I20260928 14:47:42.812009 20137 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:42.812198 20137 heartbeater.cc:511] Master 127.18.96.190:46547 requested a full tablet report, sending...
I20260928 14:47:42.812742 19933 ts_manager.cc:194] Registered new tserver with Master: 9b40a703ae0c4ae7a1a320b5e9bab7ac (127.18.96.129:45909)
I20260928 14:47:42.813665 19933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:45441
I20260928 14:47:42.816416 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 20006
I20260928 14:47:42.816540 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:47:42.816731 18818 external_mini_cluster.cc:1428] Setting key 6ef51ec1f81ad29969dca8720f2ebe60
I20260928 14:47:42.818599 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.813006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:44573 (local address 127.18.96.190:46547)
0928 14:47:42.813046 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.813047 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.813117 (+ 70us) server_negotiation.cc:407] Connection header received
0928 14:47:42.813182 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.813183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.813190 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.813229 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:42.813554 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.814203 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.814878 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.815072 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.815137 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.815979 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:42.815981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:42.815981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:42.815987 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:42.817504 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.817823 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.817824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.817825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.817879 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.818090 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.818091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.818091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.818198 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:42.818270 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.818431 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.818519 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:47:42.818608 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.812929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46547 (local address 127.18.96.129:44573)
0928 14:47:42.813058 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.813069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.813129 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.813245 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.813246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.813430 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:42.813497 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.813499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.814285 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.814285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.814780 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.814782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.815145 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.815145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.815198 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.815557 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:42.815561 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:42.815839 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:42.817580 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.817580 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.817581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.817750 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.817953 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.817954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.817955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.818018 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.818263 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:42.818264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:42.818320 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.818406 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.818506 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:47:42.842742 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:46547
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:42.922338 20146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:42.922530 20146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:42.922554 20146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:42.923764 20146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:42.923806 20146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:42.923844 20146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:47:42.923902 20146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:47:42.925401 20146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.190:46547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:42.925589 20146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:42.925760 20146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:42.931149 20152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:42.931159 20151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:42.931216 20146 server_base.cc:1034] running on GCE node
W20260928 14:47:42.931164 20154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:42.931659 20146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:42.931934 20146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:42.933123 20146 hybrid_clock.cc:648] HybridClock initialized: now 1790606862933110 us; error 27 us; skew 500 ppm
Sep 28 14:47:42 dist-test-slave-j968 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:42.935581 20146 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:47:42.936040 20146 webserver.cc:460] Webserver started at http://127.18.96.130:37365/ using document root <none> and password file <none>
I20260928 14:47:42.936245 20146 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:42.936286 20146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:42.936375 20146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:42.937389 20146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "5600caa0d6194387a26a8f7264b80bac"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-j968"
server_key: "589d4780e5d0228c73a90d40a91731cd"
server_key_iv: "125a371054025d72b70a2a7a291b42a9"
server_key_version: "encryptionkey@0"
I20260928 14:47:42.937709 20146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "5600caa0d6194387a26a8f7264b80bac"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-j968"
server_key: "589d4780e5d0228c73a90d40a91731cd"
server_key_iv: "125a371054025d72b70a2a7a291b42a9"
server_key_version: "encryptionkey@0"
I20260928 14:47:42.938935 20146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:47:42.939617 20161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.939769 20146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:47:42.939850 20146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "5600caa0d6194387a26a8f7264b80bac"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-j968"
server_key: "589d4780e5d0228c73a90d40a91731cd"
server_key_iv: "125a371054025d72b70a2a7a291b42a9"
server_key_version: "encryptionkey@0"
I20260928 14:47:42.939958 20146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:42.960531 20146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:42.962832 20146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:42.962946 20146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:42.963392 20146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:42.963426 20146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.963449 20146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:42.963461 20146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.970057 20146 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:39633
I20260928 14:47:42.970157 20276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:39633 every 8 connection(s)
I20260928 14:47:42.970477 20146 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:47:42 dist-test-slave-j968 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:42.974581 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.964101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:39221 (local address 127.18.96.190:46547)
0928 14:47:42.964173 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.964176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.964446 (+ 270us) server_negotiation.cc:407] Connection header received
0928 14:47:42.965258 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.965260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.965273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.965367 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:42.966355 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.967279 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.969031 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.969374 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.969455 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.972063 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:42.972065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:42.972065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:42.972073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:42.973392 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.973775 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.973776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.973776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.973828 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.974037 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.974038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.974038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.974149 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:42.974207 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.974442 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.974510 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:47:42.974749 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.964087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46547 (local address 127.18.96.130:39221)
0928 14:47:42.964298 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.964342 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.965139 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.965485 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.965491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.965901 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:42.966225 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.966231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.967392 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.967393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.968914 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.968917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.969476 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.969478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.969637 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.970275 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:42.970285 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:42.971973 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:42.973470 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.973477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.973478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.973708 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.973885 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.973886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.973886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.973980 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.974209 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:42.974212 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:42.974321 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.974401 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.974517 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":134,"threads_started":1}
I20260928 14:47:42.975360 20277 heartbeater.cc:347] Connected to a master server at 127.18.96.190:46547
I20260928 14:47:42.975449 20277 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:42.975670 20277 heartbeater.cc:511] Master 127.18.96.190:46547 requested a full tablet report, sending...
I20260928 14:47:42.976099 19934 ts_manager.cc:194] Registered new tserver with Master: 5600caa0d6194387a26a8f7264b80bac (127.18.96.130:39633)
I20260928 14:47:42.976581 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:39221
I20260928 14:47:42.977761 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 20146
I20260928 14:47:42.977945 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:47:42.978159 18818 external_mini_cluster.cc:1428] Setting key 72b76daacffa08a65983276a833d1be7
I20260928 14:47:42.982592 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.976568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:39583 (local address 127.18.96.190:46547)
0928 14:47:42.976608 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.976610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.976733 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:47:42.976811 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.976812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.976823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.976891 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:42.977273 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.977743 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.978793 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.979017 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.979066 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.980103 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:42.980104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:42.980104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:42.980110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:42.981451 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.981830 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.981831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.981831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.981910 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.982127 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.982128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.982128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.982248 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:42.982298 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.982475 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.982528 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:47:42.982752 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.976511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46547 (local address 127.18.96.130:39583)
0928 14:47:42.976672 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.976685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.976749 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.976889 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.976890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.977084 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:42.977185 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.977187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.977849 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.977850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.978723 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.978725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.979093 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.979094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.979179 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.979557 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:42.979563 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:42.980019 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:42.981510 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.981511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.981511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.981755 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.981987 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.981988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.981988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.982063 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.982314 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:42.982315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:42.982366 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.982459 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.982587 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:47:43.004773 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:46547
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:43.086251 20286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:43.086458 20286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:43.086483 20286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:43.087555 20286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:43.087592 20286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:43.087612 20286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:47:43.087630 20286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:47:43.088991 20286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.190:46547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:43.089176 20286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:43.089344 20286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:43.094347 20291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:43.094355 20294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:43.094447 20286 server_base.cc:1034] running on GCE node
W20260928 14:47:43.094365 20292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:43.094831 20286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:43.095053 20286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:43.096194 20286 hybrid_clock.cc:648] HybridClock initialized: now 1790606863096183 us; error 31 us; skew 500 ppm
Sep 28 14:47:43 dist-test-slave-j968 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606863, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:43.098629 20286 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:47:43.099112 20286 webserver.cc:460] Webserver started at http://127.18.96.131:40331/ using document root <none> and password file <none>
I20260928 14:47:43.099344 20286 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:43.099390 20286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:43.099519 20286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:43.100566 20286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "22ae489031bc444bb5453bb3f96c2c4f"
format_stamp: "Formatted at 2026-09-28 14:47:43 on dist-test-slave-j968"
server_key: "8a1c6be0219d4ce9d48231e66a855554"
server_key_iv: "3b06be888d26e8cbda16777b60653425"
server_key_version: "encryptionkey@0"
I20260928 14:47:43.100906 20286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "22ae489031bc444bb5453bb3f96c2c4f"
format_stamp: "Formatted at 2026-09-28 14:47:43 on dist-test-slave-j968"
server_key: "8a1c6be0219d4ce9d48231e66a855554"
server_key_iv: "3b06be888d26e8cbda16777b60653425"
server_key_version: "encryptionkey@0"
I20260928 14:47:43.102233 20286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:47:43.102958 20301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:43.103157 20286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:43.103238 20286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "22ae489031bc444bb5453bb3f96c2c4f"
format_stamp: "Formatted at 2026-09-28 14:47:43 on dist-test-slave-j968"
server_key: "8a1c6be0219d4ce9d48231e66a855554"
server_key_iv: "3b06be888d26e8cbda16777b60653425"
server_key_version: "encryptionkey@0"
I20260928 14:47:43.103289 20286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:43.114313 20286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:43.116640 20286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:43.116755 20286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:43.117223 20286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:43.117264 20286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:43.117285 20286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:43.117298 20286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:43.124341 20286 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:34279
I20260928 14:47:43.124428 20416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:34279 every 8 connection(s)
Sep 28 14:47:43 dist-test-slave-j968 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606863, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:43.124742 20286 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:47:43.128021 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.117747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:34751 (local address 127.18.96.190:46547)
0928 14:47:43.117822 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.117825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.118069 (+ 244us) server_negotiation.cc:407] Connection header received
0928 14:47:43.118698 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.118699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.118712 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.118774 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:43.119766 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.120603 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.122330 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.122528 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.122630 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.125346 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:43.125348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:43.125348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:43.125356 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:43.126580 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.126994 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.126995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.126996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.127058 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.127320 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.127321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.127321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.127446 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:43.127525 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.127803 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.127929 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:47:43.128288 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.117719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46547 (local address 127.18.96.131:34751)
0928 14:47:43.117889 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.117929 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.118553 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.118890 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.118895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.119338 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:43.119656 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.119664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.120708 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.120709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.122212 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.122216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.122652 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.122653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.122806 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.123460 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:43.123470 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:43.125262 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:43.126669 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:43.126676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:43.126678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:43.126896 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:43.127140 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:43.127140 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:43.127141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:43.127250 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:43.127528 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:43.127535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:43.127673 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.127781 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.127990 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260928 14:47:43.128760 20417 heartbeater.cc:347] Connected to a master server at 127.18.96.190:46547
I20260928 14:47:43.128841 20417 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:43.129026 20417 heartbeater.cc:511] Master 127.18.96.190:46547 requested a full tablet report, sending...
I20260928 14:47:43.129359 19934 ts_manager.cc:194] Registered new tserver with Master: 22ae489031bc444bb5453bb3f96c2c4f (127.18.96.131:34279)
I20260928 14:47:43.129557 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 20286
I20260928 14:47:43.129645 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:47:43.129815 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:34751
I20260928 14:47:43.129840 18818 external_mini_cluster.cc:1428] Setting key a03641ca0bb766c3fea81bcc40af7f7e
I20260928 14:47:43.130760 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:47:43.136765 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.130017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:41097 (local address 127.18.96.190:46547)
0928 14:47:43.130098 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.130100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.130214 (+ 114us) server_negotiation.cc:407] Connection header received
0928 14:47:43.130375 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.130376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.130386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.130440 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:43.130794 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.131493 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.132645 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.132961 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.133040 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.133940 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:43.133941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:43.133942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:43.133949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:43.135585 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.136030 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.136031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.136032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.136090 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.136298 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.136299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.136299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.136414 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:43.136465 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.136642 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.136701 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:47:43.136955 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.129844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46547 (local address 127.18.96.131:41097)
0928 14:47:43.130135 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.130145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.130297 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.130445 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.130446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.130632 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:43.130688 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.130689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.131640 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.131641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.132544 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.132547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.133047 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.133048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.133107 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.133535 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:43.133539 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:43.133850 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:43.135667 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:43.135668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:43.135668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:43.135947 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:43.136151 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:43.136152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:43.136153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:43.136222 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:43.136483 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:43.136484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:43.136545 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.136618 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.136734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":73,"threads_started":1}
Sep 28 14:47:43 dist-test-slave-j968 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:47:43 dist-test-slave-j968 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:43.155165 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.148406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.18.96.190:46547)
0928 14:47:43.148474 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.148475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.148484 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:47:43.148507 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.148508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.148516 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.148564 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:43.149127 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.149609 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.150301 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.150474 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.150520 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.152784 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:43.152785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:43.152786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:43.152792 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:43.154066 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.154382 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.154383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.154384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.154438 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.154671 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.154671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.154672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.154789 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:43.154935 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.154964 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.155030 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:47:43.156920 19934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:47:43.158428 19934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:43.164875 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.160637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39633 (local address 127.0.0.1:42612)
0928 14:47:43.160820 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.160866 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.160926 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.161440 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.161443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.161445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:43.161552 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.161554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.162463 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.162464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.163654 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.163656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.164587 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.164588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.164666 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.164680 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.164744 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.164796 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":30,"thread_start_us":60,"threads_started":1}
I20260928 14:47:43.165035 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.160825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:34279 (local address 127.0.0.1:44870)
0928 14:47:43.161015 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.161023 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.161061 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.161442 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.161443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.161448 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:43.161602 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.161604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.162851 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.162852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.163854 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.163856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.164755 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.164756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.164824 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.164828 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.164885 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.164930 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":73,"thread_start_us":47,"threads_started":1}
I20260928 14:47:43.165217 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.160740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.18.96.130:39633)
0928 14:47:43.161205 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.161211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.161222 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:47:43.161282 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.161284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.161300 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.161368 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:43.161646 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.162359 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.163726 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.164488 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.164566 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.164625 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.165080 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.165126 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":99,"threads_started":1}
I20260928 14:47:43.165843 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.160637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:45909 (local address 127.0.0.1:50266)
0928 14:47:43.161003 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.161015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.161072 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.161460 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.161460 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.161462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:43.161540 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.161542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.163938 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.163939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.164682 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.164684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.165583 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.165584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.165661 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.165664 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.165722 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.165761 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":4,"thread_start_us":57,"threads_started":1}
I20260928 14:47:43.165865 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.160838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.18.96.129:45909)
0928 14:47:43.161228 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.161232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.161240 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:47:43.161291 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.161293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.161309 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.161388 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:43.161622 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.162752 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.164779 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.165487 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.165576 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.165627 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.165735 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.165772 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":80,"threads_started":1}
I20260928 14:47:43.165932 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 0ebf81d465c04fa59e59f30b2106b72e (DEFAULT_TABLE table=test-table [id=8d63fbacf5fb44ccaa6fa4abb16a8faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:43.166378 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ebf81d465c04fa59e59f30b2106b72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:43.166507 20071 tablet_service.cc:1467] Processing CreateTablet for tablet 0ebf81d465c04fa59e59f30b2106b72e (DEFAULT_TABLE table=test-table [id=8d63fbacf5fb44ccaa6fa4abb16a8faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:43.166848 20071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ebf81d465c04fa59e59f30b2106b72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:43.168162 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.161047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.18.96.131:34279)
0928 14:47:43.161216 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.161220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.161229 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:47:43.161266 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.161269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.161285 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.161363 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:43.161668 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.162757 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.163962 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.164677 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.164746 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.164796 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.164899 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.168044 (+ 3145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260928 14:47:43.169446 20437 tablet_bootstrap.cc:492] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: Bootstrap starting.
I20260928 14:47:43.170018 20336 tablet_service.cc:1467] Processing CreateTablet for tablet 0ebf81d465c04fa59e59f30b2106b72e (DEFAULT_TABLE table=test-table [id=8d63fbacf5fb44ccaa6fa4abb16a8faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:43.170149 20437 tablet_bootstrap.cc:654] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:43.170368 20336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ebf81d465c04fa59e59f30b2106b72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:43.170518 20437 log.cc:824] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:43.171231 20437 tablet_bootstrap.cc:492] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: No bootstrap required, opened a new log
I20260928 14:47:43.171314 20437 ts_tablet_manager.cc:1397] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:47:43.172748 20437 raft_consensus.cc:348] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.172914 20437 raft_consensus.cc:374] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:43.172958 20437 raft_consensus.cc:729] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5600caa0d6194387a26a8f7264b80bac, State: Initialized, Role: FOLLOWER
I20260928 14:47:43.172999 20440 tablet_bootstrap.cc:492] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: Bootstrap starting.
I20260928 14:47:43.173064 20437 consensus_queue.cc:260] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.173292 20438 tablet_bootstrap.cc:492] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Bootstrap starting.
I20260928 14:47:43.173332 20437 ts_tablet_manager.cc:1428] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:47:43.173445 20277 heartbeater.cc:503] Master 127.18.96.190:46547 was elected leader, sending a full tablet report...
I20260928 14:47:43.173975 20440 tablet_bootstrap.cc:654] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:43.173985 20438 tablet_bootstrap.cc:654] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:43.174309 20440 log.cc:824] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:43.174309 20438 log.cc:824] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:43.175134 20440 tablet_bootstrap.cc:492] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: No bootstrap required, opened a new log
I20260928 14:47:43.175173 20438 tablet_bootstrap.cc:492] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: No bootstrap required, opened a new log
I20260928 14:47:43.175217 20440 ts_tablet_manager.cc:1397] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:47:43.175240 20438 ts_tablet_manager.cc:1397] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:47:43.176752 20440 raft_consensus.cc:348] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.176898 20440 raft_consensus.cc:374] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:43.176811 20438 raft_consensus.cc:348] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.176940 20440 raft_consensus.cc:729] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ae489031bc444bb5453bb3f96c2c4f, State: Initialized, Role: FOLLOWER
I20260928 14:47:43.176945 20438 raft_consensus.cc:374] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:43.176986 20438 raft_consensus.cc:729] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b40a703ae0c4ae7a1a320b5e9bab7ac, State: Initialized, Role: FOLLOWER
I20260928 14:47:43.177057 20440 consensus_queue.cc:260] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.177126 20438 consensus_queue.cc:260] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.177357 20440 ts_tablet_manager.cc:1428] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:47:43.177376 20417 heartbeater.cc:503] Master 127.18.96.190:46547 was elected leader, sending a full tablet report...
I20260928 14:47:43.177496 20137 heartbeater.cc:503] Master 127.18.96.190:46547 was elected leader, sending a full tablet report...
I20260928 14:47:43.177515 20438 ts_tablet_manager.cc:1428] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260928 14:47:43.221136 20278 tablet.cc:2369] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:47:43.309440 20138 tablet.cc:2369] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:47:43.355852 20445 raft_consensus.cc:482] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:43.355999 20445 raft_consensus.cc:504] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.356400 20445 leader_election.cc:288] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5600caa0d6194387a26a8f7264b80bac (127.18.96.130:39633), 22ae489031bc444bb5453bb3f96c2c4f (127.18.96.131:34279)
I20260928 14:47:43.359373 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.356753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:35403 (local address 127.18.96.130:39633)
0928 14:47:43.356801 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.356803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.356838 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:47:43.356927 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.356928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.356937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.357014 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:43.357176 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.357650 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.358646 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.358997 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.359101 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.359110 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.359260 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.359295 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:47:43.359422 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.356622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39633 (local address 127.18.96.129:35403)
0928 14:47:43.356757 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.356770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.356849 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.357021 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.357022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.357024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:43.357100 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.357102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.357734 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.357735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.358521 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.358523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.359106 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.359107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.359183 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.359187 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.359253 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.359295 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260928 14:47:43.359416 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.356790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:50093 (local address 127.18.96.131:34279)
0928 14:47:43.356834 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.356836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.356845 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:47:43.356946 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.356946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.356954 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.357021 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:43.357192 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.357670 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.358665 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.359051 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.359140 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.359149 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.359297 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.359337 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:47:43.359436 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.356623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:34279 (local address 127.18.96.129:50093)
0928 14:47:43.356760 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.356773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.356829 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.357027 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.357028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.357030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:43.357116 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.357118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.357756 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.357757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.358561 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.358563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.359138 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.359139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.359222 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.359225 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.359291 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.359332 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":113,"thread_start_us":65,"threads_started":1}
I20260928 14:47:43.359804 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ebf81d465c04fa59e59f30b2106b72e" candidate_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ae489031bc444bb5453bb3f96c2c4f" is_pre_election: true
I20260928 14:47:43.359944 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ebf81d465c04fa59e59f30b2106b72e" candidate_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600caa0d6194387a26a8f7264b80bac" is_pre_election: true
I20260928 14:47:43.360000 20371 raft_consensus.cc:2454] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b40a703ae0c4ae7a1a320b5e9bab7ac in term 0.
I20260928 14:47:43.360113 20231 raft_consensus.cc:2454] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b40a703ae0c4ae7a1a320b5e9bab7ac in term 0.
I20260928 14:47:43.360280 20025 leader_election.cc:302] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ae489031bc444bb5453bb3f96c2c4f, 9b40a703ae0c4ae7a1a320b5e9bab7ac; no voters:
I20260928 14:47:43.360433 20445 raft_consensus.cc:2781] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:43.360491 20445 raft_consensus.cc:482] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:43.360522 20445 raft_consensus.cc:3037] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:43.361177 20445 raft_consensus.cc:504] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.361321 20445 leader_election.cc:288] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [CANDIDATE]: Term 1 election: Requested vote from peers 5600caa0d6194387a26a8f7264b80bac (127.18.96.130:39633), 22ae489031bc444bb5453bb3f96c2c4f (127.18.96.131:34279)
I20260928 14:47:43.361472 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ebf81d465c04fa59e59f30b2106b72e" candidate_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ae489031bc444bb5453bb3f96c2c4f"
I20260928 14:47:43.361469 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ebf81d465c04fa59e59f30b2106b72e" candidate_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600caa0d6194387a26a8f7264b80bac"
I20260928 14:47:43.361564 20231 raft_consensus.cc:3037] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:43.361574 20371 raft_consensus.cc:3037] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:43.362248 20231 raft_consensus.cc:2454] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b40a703ae0c4ae7a1a320b5e9bab7ac in term 1.
I20260928 14:47:43.362327 20371 raft_consensus.cc:2454] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b40a703ae0c4ae7a1a320b5e9bab7ac in term 1.
I20260928 14:47:43.362425 20022 leader_election.cc:302] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5600caa0d6194387a26a8f7264b80bac, 9b40a703ae0c4ae7a1a320b5e9bab7ac; no voters:
I20260928 14:47:43.362533 20445 raft_consensus.cc:2781] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:43.362648 20445 raft_consensus.cc:686] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 LEADER]: Becoming Leader. State: Replica: 9b40a703ae0c4ae7a1a320b5e9bab7ac, State: Running, Role: LEADER
I20260928 14:47:43.362766 20445 consensus_queue.cc:237] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.363579 19933 catalog_manager.cc:5495] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b40a703ae0c4ae7a1a320b5e9bab7ac (127.18.96.129). New cstate: current_term: 1 leader_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } health_report { overall_health: UNKNOWN } } }
W20260928 14:47:43.375319 20418 tablet.cc:2369] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:47:43.393144 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.390652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.18.96.129:45909)
0928 14:47:43.390700 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.390703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.390710 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:47:43.390799 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.390800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.390817 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.390866 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:43.391013 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.391473 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.392306 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.392485 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.392537 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.392775 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:43.392874 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:43.392932 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.393050 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.393071 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:47:43.394961 20371 raft_consensus.cc:1261] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [term 1 FOLLOWER]: Refusing update from remote peer 9b40a703ae0c4ae7a1a320b5e9bab7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:43.394982 20231 raft_consensus.cc:1261] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Refusing update from remote peer 9b40a703ae0c4ae7a1a320b5e9bab7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:43.395309 20448 consensus_queue.cc:1035] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:43.395450 20445 consensus_queue.cc:1035] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:43.398000 20453 mvcc.cc:205] Tried to move back new op lower bound from 7334325712462897152 to 7334325712334454784. Current Snapshot: MvccSnapshot[applied={T|T < 7334325712462897152}]
I20260928 14:47:43.401183 19934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:43.401265 19934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:43.402079 19934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd: Sending DeleteTablet for 3 replicas of tablet 0ebf81d465c04fa59e59f30b2106b72e
I20260928 14:47:43.402395 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ebf81d465c04fa59e59f30b2106b72e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:43 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:42612
I20260928 14:47:43.402395 20336 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ebf81d465c04fa59e59f30b2106b72e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:43 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:44870
I20260928 14:47:43.402662 20461 tablet_replica.cc:330] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: stopping tablet replica
I20260928 14:47:43.402663 20071 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ebf81d465c04fa59e59f30b2106b72e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:43 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:50266
I20260928 14:47:43.402767 20461 raft_consensus.cc:2229] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:43.402879 20461 raft_consensus.cc:2258] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:43.402844 19934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:47:43.402992 19934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:43.402885 20463 tablet_replica.cc:330] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: stopping tablet replica
I20260928 14:47:43.403215 20463 raft_consensus.cc:2229] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:47:43.403326 20461 ts_tablet_manager.cc:1905] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:43.403357 20463 raft_consensus.cc:2258] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:43.403791 20463 ts_tablet_manager.cc:1905] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:43.404536 20336 tablet_service.cc:1467] Processing CreateTablet for tablet 463b85605bc6483aac4aa4bf27363850 (DEFAULT_TABLE table=test-table [id=8e5fb3b4e3ef4826a40aee20972b3141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:43.404754 20336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463b85605bc6483aac4aa4bf27363850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:43.405071 20462 tablet_replica.cc:330] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: stopping tablet replica
I20260928 14:47:43.405158 20462 raft_consensus.cc:2229] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:43.405241 20462 raft_consensus.cc:2258] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:43.405545 20463 ts_tablet_manager.cc:1918] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:47:43.405614 20463 log.cc:1196] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/0ebf81d465c04fa59e59f30b2106b72e
I20260928 14:47:43.405869 20463 ts_tablet_manager.cc:1939] T 0ebf81d465c04fa59e59f30b2106b72e P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Deleting consensus metadata
I20260928 14:47:43.405967 20440 tablet_bootstrap.cc:492] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: Bootstrap starting.
I20260928 14:47:43.406028 20461 ts_tablet_manager.cc:1918] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:47:43.406085 20461 log.cc:1196] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/0ebf81d465c04fa59e59f30b2106b72e
I20260928 14:47:43.406340 20461 ts_tablet_manager.cc:1939] T 0ebf81d465c04fa59e59f30b2106b72e P 22ae489031bc444bb5453bb3f96c2c4f: Deleting consensus metadata
I20260928 14:47:43.406340 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 463b85605bc6483aac4aa4bf27363850 (DEFAULT_TABLE table=test-table [id=8e5fb3b4e3ef4826a40aee20972b3141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:43.406386 19921 catalog_manager.cc:4841] TS 9b40a703ae0c4ae7a1a320b5e9bab7ac (127.18.96.129:45909): tablet 0ebf81d465c04fa59e59f30b2106b72e (table test-table [id=8d63fbacf5fb44ccaa6fa4abb16a8faf]) successfully deleted
I20260928 14:47:43.406415 20440 tablet_bootstrap.cc:654] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:43.406454 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463b85605bc6483aac4aa4bf27363850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:43.406533 20071 tablet_service.cc:1467] Processing CreateTablet for tablet 463b85605bc6483aac4aa4bf27363850 (DEFAULT_TABLE table=test-table [id=8e5fb3b4e3ef4826a40aee20972b3141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:43.406626 20071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463b85605bc6483aac4aa4bf27363850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:43.407547 20438 tablet_bootstrap.cc:492] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Bootstrap starting.
I20260928 14:47:43.407841 19921 catalog_manager.cc:4841] TS 22ae489031bc444bb5453bb3f96c2c4f (127.18.96.131:34279): tablet 0ebf81d465c04fa59e59f30b2106b72e (table test-table [id=8d63fbacf5fb44ccaa6fa4abb16a8faf]) successfully deleted
I20260928 14:47:43.407893 20437 tablet_bootstrap.cc:492] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: Bootstrap starting.
I20260928 14:47:43.408017 20438 tablet_bootstrap.cc:654] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:43.408277 20437 tablet_bootstrap.cc:654] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:43.408679 20438 tablet_bootstrap.cc:492] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: No bootstrap required, opened a new log
I20260928 14:47:43.408728 20438 ts_tablet_manager.cc:1397] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:43.408815 20462 ts_tablet_manager.cc:1905] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:43.408895 20438 raft_consensus.cc:348] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.408963 20438 raft_consensus.cc:374] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:43.408994 20438 raft_consensus.cc:729] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b40a703ae0c4ae7a1a320b5e9bab7ac, State: Initialized, Role: FOLLOWER
I20260928 14:47:43.409037 20438 consensus_queue.cc:260] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.409121 20438 ts_tablet_manager.cc:1428] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:43.409590 20437 tablet_bootstrap.cc:492] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: No bootstrap required, opened a new log
I20260928 14:47:43.409637 20437 ts_tablet_manager.cc:1397] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:47:43.409783 20437 raft_consensus.cc:348] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.409878 20437 raft_consensus.cc:374] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:43.409904 20437 raft_consensus.cc:729] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5600caa0d6194387a26a8f7264b80bac, State: Initialized, Role: FOLLOWER
I20260928 14:47:43.409945 20437 consensus_queue.cc:260] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.410025 20437 ts_tablet_manager.cc:1428] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:43.410334 20462 ts_tablet_manager.cc:1918] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:47:43.410436 20462 log.cc:1196] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/0ebf81d465c04fa59e59f30b2106b72e
I20260928 14:47:43.410717 20462 ts_tablet_manager.cc:1939] T 0ebf81d465c04fa59e59f30b2106b72e P 5600caa0d6194387a26a8f7264b80bac: Deleting consensus metadata
I20260928 14:47:43.411087 19918 catalog_manager.cc:4841] TS 5600caa0d6194387a26a8f7264b80bac (127.18.96.130:39633): tablet 0ebf81d465c04fa59e59f30b2106b72e (table test-table [id=8d63fbacf5fb44ccaa6fa4abb16a8faf]) successfully deleted
I20260928 14:47:43.416225 20440 tablet_bootstrap.cc:492] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: No bootstrap required, opened a new log
I20260928 14:47:43.416296 20440 ts_tablet_manager.cc:1397] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260928 14:47:43.416447 20440 raft_consensus.cc:348] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.416527 20440 raft_consensus.cc:374] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:43.416559 20440 raft_consensus.cc:729] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ae489031bc444bb5453bb3f96c2c4f, State: Initialized, Role: FOLLOWER
I20260928 14:47:43.416611 20440 consensus_queue.cc:260] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.416703 20440 ts_tablet_manager.cc:1428] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:43.471005 20444 raft_consensus.cc:482] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:43.471098 20444 raft_consensus.cc:504] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.471438 20444 leader_election.cc:288] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b40a703ae0c4ae7a1a320b5e9bab7ac (127.18.96.129:45909), 5600caa0d6194387a26a8f7264b80bac (127.18.96.130:39633)
I20260928 14:47:43.474822 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.471780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:53237 (local address 127.18.96.129:45909)
0928 14:47:43.471841 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.471843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.471851 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:47:43.471897 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.471898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.471909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.471969 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:43.472118 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.472851 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.474102 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.474451 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.474540 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.474549 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.474722 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.474760 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:47:43.474840 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.471574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:45909 (local address 127.18.96.131:53237)
0928 14:47:43.471665 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.471678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.471720 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.471976 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.471977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.471978 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:43.472049 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.472051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.472939 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.472940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.474011 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.474013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.474537 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.474538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.474622 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.474628 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.474709 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.474749 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:47:43.475158 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.471798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:41625 (local address 127.18.96.130:39633)
0928 14:47:43.471844 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.471846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.471857 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:47:43.471948 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.471949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.471958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.472011 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:43.472158 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.472643 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.474100 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.474668 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.474746 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.474755 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.475049 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.475094 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:47:43.475201 20091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463b85605bc6483aac4aa4bf27363850" candidate_uuid: "22ae489031bc444bb5453bb3f96c2c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" is_pre_election: true
I20260928 14:47:43.475322 20091 raft_consensus.cc:2454] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ae489031bc444bb5453bb3f96c2c4f in term 0.
I20260928 14:47:43.475423 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.471641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39633 (local address 127.18.96.131:41625)
0928 14:47:43.471810 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.471819 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.471860 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.472028 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.472029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.472030 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:43.472098 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.472099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.472736 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.472737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.473993 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.473996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.474851 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.474852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.474927 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.474932 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.475270 (+ 338us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.475313 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":58,"threads_started":1}
I20260928 14:47:43.475526 20305 leader_election.cc:302] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ae489031bc444bb5453bb3f96c2c4f, 9b40a703ae0c4ae7a1a320b5e9bab7ac; no voters:
I20260928 14:47:43.475641 20444 raft_consensus.cc:2781] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:43.475668 20444 raft_consensus.cc:482] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:43.475627 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463b85605bc6483aac4aa4bf27363850" candidate_uuid: "22ae489031bc444bb5453bb3f96c2c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600caa0d6194387a26a8f7264b80bac" is_pre_election: true
I20260928 14:47:43.475687 20444 raft_consensus.cc:3037] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:43.475711 20231 raft_consensus.cc:2454] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ae489031bc444bb5453bb3f96c2c4f in term 0.
I20260928 14:47:43.476514 20444 raft_consensus.cc:504] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.476662 20444 leader_election.cc:288] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [CANDIDATE]: Term 1 election: Requested vote from peers 9b40a703ae0c4ae7a1a320b5e9bab7ac (127.18.96.129:45909), 5600caa0d6194387a26a8f7264b80bac (127.18.96.130:39633)
I20260928 14:47:43.476809 20091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463b85605bc6483aac4aa4bf27363850" candidate_uuid: "22ae489031bc444bb5453bb3f96c2c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac"
I20260928 14:47:43.476884 20091 raft_consensus.cc:3037] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:43.476857 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463b85605bc6483aac4aa4bf27363850" candidate_uuid: "22ae489031bc444bb5453bb3f96c2c4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600caa0d6194387a26a8f7264b80bac"
I20260928 14:47:43.476917 20231 raft_consensus.cc:3037] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:43.477628 20091 raft_consensus.cc:2454] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ae489031bc444bb5453bb3f96c2c4f in term 1.
I20260928 14:47:43.477625 20231 raft_consensus.cc:2454] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ae489031bc444bb5453bb3f96c2c4f in term 1.
I20260928 14:47:43.477830 20302 leader_election.cc:302] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ae489031bc444bb5453bb3f96c2c4f, 5600caa0d6194387a26a8f7264b80bac; no voters:
I20260928 14:47:43.477946 20444 raft_consensus.cc:2781] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:43.478215 20444 raft_consensus.cc:686] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 1 LEADER]: Becoming Leader. State: Replica: 22ae489031bc444bb5453bb3f96c2c4f, State: Running, Role: LEADER
I20260928 14:47:43.478308 20444 consensus_queue.cc:237] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.479068 19934 catalog_manager.cc:5495] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ae489031bc444bb5453bb3f96c2c4f (127.18.96.131). New cstate: current_term: 1 leader_uuid: "22ae489031bc444bb5453bb3f96c2c4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } health_report { overall_health: HEALTHY } } }
Sep 28 14:47:43 dist-test-slave-j968 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:43.508198 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.500589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46547 (local address 127.0.0.1:45928)
0928 14:47:43.500739 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.500751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.500816 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.501015 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.501016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.501290 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:43.501338 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.501340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.502299 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.502300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.502809 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.502810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.503256 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.503256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.503294 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.503829 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:43.503833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:43.505469 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:43.507184 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:43.507189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:43.507191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:43.507371 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:43.507573 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:43.507574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:43.507574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:43.507627 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:43.507911 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:43.507918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:43.507973 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.508029 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.508116 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260928 14:47:43.508299 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.500658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.18.96.190:46547)
0928 14:47:43.500710 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.500712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.500827 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:47:43.500892 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.500893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.500907 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.500980 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:43.501424 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.502207 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.502891 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.503175 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.503254 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.505558 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:43.505560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:43.505560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:43.505567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:43.507095 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.507443 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.507444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.507445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.507508 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.507687 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.507688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.507688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.507814 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:43.507905 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.508071 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.508177 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":28}
I20260928 14:47:43.510596 19934 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:45928:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:47:43.510816 19934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:43.512254 20211 tablet_service.cc:1467] Processing CreateTablet for tablet db920925a5ca4f96a72f7deae8c3971c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d6f1af648d94e68b3bb32c37d1fb96e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:47:43.512253 20336 tablet_service.cc:1467] Processing CreateTablet for tablet db920925a5ca4f96a72f7deae8c3971c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d6f1af648d94e68b3bb32c37d1fb96e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:47:43.512404 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db920925a5ca4f96a72f7deae8c3971c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:43.512404 20336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db920925a5ca4f96a72f7deae8c3971c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:43.513921 20437 tablet_bootstrap.cc:492] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: Bootstrap starting.
I20260928 14:47:43.513932 20440 tablet_bootstrap.cc:492] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: Bootstrap starting.
I20260928 14:47:43.513985 20071 tablet_service.cc:1467] Processing CreateTablet for tablet db920925a5ca4f96a72f7deae8c3971c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7d6f1af648d94e68b3bb32c37d1fb96e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:47:43.514113 20071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db920925a5ca4f96a72f7deae8c3971c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:43.514290 20437 tablet_bootstrap.cc:654] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:43.514333 20440 tablet_bootstrap.cc:654] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:43.515280 20440 tablet_bootstrap.cc:492] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: No bootstrap required, opened a new log
I20260928 14:47:43.515329 20440 ts_tablet_manager.cc:1397] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:43.515468 20440 raft_consensus.cc:348] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.515535 20440 raft_consensus.cc:374] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:43.515554 20440 raft_consensus.cc:729] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ae489031bc444bb5453bb3f96c2c4f, State: Initialized, Role: FOLLOWER
I20260928 14:47:43.515587 20440 consensus_queue.cc:260] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.515672 20440 ts_tablet_manager.cc:1428] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:43.515681 20438 tablet_bootstrap.cc:492] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Bootstrap starting.
I20260928 14:47:43.515771 20444 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.515820 20444 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:43.516027 20438 tablet_bootstrap.cc:654] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:43.516562 20438 tablet_bootstrap.cc:492] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: No bootstrap required, opened a new log
I20260928 14:47:43.516628 20438 ts_tablet_manager.cc:1397] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:43.516743 20438 raft_consensus.cc:348] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.516812 20438 raft_consensus.cc:374] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:43.516836 20438 raft_consensus.cc:729] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b40a703ae0c4ae7a1a320b5e9bab7ac, State: Initialized, Role: FOLLOWER
I20260928 14:47:43.516877 20438 consensus_queue.cc:260] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.516952 20438 ts_tablet_manager.cc:1428] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:43.516983 20458 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.517089 20458 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:43.517472 20437 tablet_bootstrap.cc:492] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: No bootstrap required, opened a new log
I20260928 14:47:43.517519 20437 ts_tablet_manager.cc:1397] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:47:43.517652 20437 raft_consensus.cc:348] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.517706 20437 raft_consensus.cc:374] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:43.517724 20437 raft_consensus.cc:729] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5600caa0d6194387a26a8f7264b80bac, State: Initialized, Role: FOLLOWER
I20260928 14:47:43.517766 20437 consensus_queue.cc:260] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.517839 20437 ts_tablet_manager.cc:1428] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:43.517946 20441 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.518054 20441 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:43.746745 20458 raft_consensus.cc:482] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:43.746848 20458 raft_consensus.cc:504] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.747107 20458 leader_election.cc:288] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5600caa0d6194387a26a8f7264b80bac (127.18.96.130:39633), 22ae489031bc444bb5453bb3f96c2c4f (127.18.96.131:34279)
I20260928 14:47:43.747335 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db920925a5ca4f96a72f7deae8c3971c" candidate_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600caa0d6194387a26a8f7264b80bac" is_pre_election: true
I20260928 14:47:43.747335 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db920925a5ca4f96a72f7deae8c3971c" candidate_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ae489031bc444bb5453bb3f96c2c4f" is_pre_election: true
I20260928 14:47:43.747447 20231 raft_consensus.cc:2454] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b40a703ae0c4ae7a1a320b5e9bab7ac in term 0.
I20260928 14:47:43.747464 20371 raft_consensus.cc:2454] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b40a703ae0c4ae7a1a320b5e9bab7ac in term 0.
I20260928 14:47:43.747591 20022 leader_election.cc:302] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5600caa0d6194387a26a8f7264b80bac, 9b40a703ae0c4ae7a1a320b5e9bab7ac; no voters:
I20260928 14:47:43.747684 20458 raft_consensus.cc:2781] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:43.747741 20458 raft_consensus.cc:482] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:43.747774 20458 raft_consensus.cc:3037] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:43.748678 20458 raft_consensus.cc:504] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.748824 20458 leader_election.cc:288] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [CANDIDATE]: Term 1 election: Requested vote from peers 5600caa0d6194387a26a8f7264b80bac (127.18.96.130:39633), 22ae489031bc444bb5453bb3f96c2c4f (127.18.96.131:34279)
I20260928 14:47:43.748960 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db920925a5ca4f96a72f7deae8c3971c" candidate_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ae489031bc444bb5453bb3f96c2c4f"
I20260928 14:47:43.748956 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db920925a5ca4f96a72f7deae8c3971c" candidate_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600caa0d6194387a26a8f7264b80bac"
I20260928 14:47:43.749035 20371 raft_consensus.cc:3037] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:43.749035 20231 raft_consensus.cc:3037] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:43.749765 20231 raft_consensus.cc:2454] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b40a703ae0c4ae7a1a320b5e9bab7ac in term 1.
I20260928 14:47:43.749765 20371 raft_consensus.cc:2454] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b40a703ae0c4ae7a1a320b5e9bab7ac in term 1.
I20260928 14:47:43.749974 20022 leader_election.cc:302] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5600caa0d6194387a26a8f7264b80bac, 9b40a703ae0c4ae7a1a320b5e9bab7ac; no voters:
I20260928 14:47:43.750087 20458 raft_consensus.cc:2781] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:43.750151 20458 raft_consensus.cc:686] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 LEADER]: Becoming Leader. State: Replica: 9b40a703ae0c4ae7a1a320b5e9bab7ac, State: Running, Role: LEADER
I20260928 14:47:43.750208 20458 consensus_queue.cc:237] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } }
I20260928 14:47:43.750487 20448 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: TxnStatusTablet state changed. Reason: New leader 9b40a703ae0c4ae7a1a320b5e9bab7ac. Latest consensus state: current_term: 1 leader_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.750592 20448 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:47:43.750782 20480 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:47:43.750912 20480 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:47:43.750847 19934 catalog_manager.cc:5495] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b40a703ae0c4ae7a1a320b5e9bab7ac (127.18.96.129). New cstate: current_term: 1 leader_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } health_report { overall_health: UNKNOWN } } }
I20260928 14:47:43.800261 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.797527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.18.96.129:45909)
0928 14:47:43.797570 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.797572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.797617 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:47:43.797702 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.797703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.797711 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.797761 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:43.797938 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.798406 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.799477 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.799660 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.799716 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.800001 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:43.800040 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:43.800084 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.800171 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.800198 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:47:43.800302 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.797469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:45909 (local address 127.0.0.1:50276)
0928 14:47:43.797564 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.797575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.797630 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.797789 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.797790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.797791 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:47:43.797845 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.797848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.798487 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.798488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.799372 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.799375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.799738 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.799739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.799808 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.799854 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:47:43.800105 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:47:43.800114 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.800167 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.800220 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 14:47:43.801334 20371 raft_consensus.cc:1261] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f [term 1 FOLLOWER]: Refusing update from remote peer 9b40a703ae0c4ae7a1a320b5e9bab7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:43.801568 20231 raft_consensus.cc:1261] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Refusing update from remote peer 9b40a703ae0c4ae7a1a320b5e9bab7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:43.801640 20458 consensus_queue.cc:1035] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:43.801780 20448 consensus_queue.cc:1035] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:43.802245 20444 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: TxnStatusTablet state changed. Reason: New leader 9b40a703ae0c4ae7a1a320b5e9bab7ac. Latest consensus state: current_term: 1 leader_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.802337 20444 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:43.802661 20441 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: TxnStatusTablet state changed. Reason: New leader 9b40a703ae0c4ae7a1a320b5e9bab7ac. Latest consensus state: current_term: 1 leader_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.802762 20441 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:43.803153 20458 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.803236 20458 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:47:43.803301 20451 mvcc.cc:205] Tried to move back new op lower bound from 7334325714128617472 to 7334325713921015808. Current Snapshot: MvccSnapshot[applied={T|T < 7334325714128617472 or (T in {7334325714128617472})}]
I20260928 14:47:43.803385 20452 mvcc.cc:205] Tried to move back new op lower bound from 7334325714128617472 to 7334325713921015808. Current Snapshot: MvccSnapshot[applied={T|T < 7334325714128617472 or (T in {7334325714128617472})}]
I20260928 14:47:43.803457 20444 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.803534 20444 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 22ae489031bc444bb5453bb3f96c2c4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:43.804210 20458 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.804291 20458 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 9b40a703ae0c4ae7a1a320b5e9bab7ac: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:47:43.804726 20441 tablet_replica.cc:439] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 } } peers { permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 } } peers { permanent_uuid: "22ae489031bc444bb5453bb3f96c2c4f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34279 } } }
I20260928 14:47:43.804813 20441 tablet_replica.cc:442] T db920925a5ca4f96a72f7deae8c3971c P 5600caa0d6194387a26a8f7264b80bac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:43.808044 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.805361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.18.96.131:34279)
0928 14:47:43.805436 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.805437 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.805445 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:47:43.805481 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.805481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.805521 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.805594 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:43.805775 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.806347 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.807165 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.807377 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.807461 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.807640 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:43.807764 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:43.807857 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.807949 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.807968 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:47:43.813544 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.810434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:45909 (local address 127.18.96.131:52791)
0928 14:47:43.810606 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.810619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.810671 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.810845 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.810845 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.810847 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:47:43.810902 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.810903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.811912 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.811914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.812517 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.812519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.813116 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.813117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.813192 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.813204 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:47:43.813407 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:47:43.813408 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.813440 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.813475 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260928 14:47:43.813539 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.810615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:52791 (local address 127.18.96.129:45909)
0928 14:47:43.810655 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.810657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.810666 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:47:43.810762 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.810763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.810776 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.810839 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:43.810971 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.811667 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.812637 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.812905 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.813009 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.813303 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:43.813339 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:43.813408 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.813447 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.813470 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:47:43.816279 20231 raft_consensus.cc:1261] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Refusing update from remote peer 22ae489031bc444bb5453bb3f96c2c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:43.816527 20091 raft_consensus.cc:1261] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 FOLLOWER]: Refusing update from remote peer 22ae489031bc444bb5453bb3f96c2c4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:43.816529 20468 consensus_queue.cc:1035] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5600caa0d6194387a26a8f7264b80bac" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:43.817078 20468 consensus_queue.cc:1035] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b40a703ae0c4ae7a1a320b5e9bab7ac" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:43.825578 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.822643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:33557 (local address 127.18.96.131:34279)
0928 14:47:43.822689 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.822691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.822826 (+ 135us) server_negotiation.cc:407] Connection header received
0928 14:47:43.822929 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.822930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.822940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.822969 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:43.823157 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.823668 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.824796 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.825071 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.825119 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.825353 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:43.825381 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:43.825426 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.825500 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.825517 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:47:43.825620 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.822602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:34279 (local address 127.18.96.129:33557)
0928 14:47:43.822757 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:43.822769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:43.822855 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:43.823031 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:43.823032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:43.823033 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:47:43.823087 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.823088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.823758 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.823759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.824685 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:43.824688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.825179 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.825180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:43.825249 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.825265 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:47:43.825430 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:47:43.825431 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:47:43.825491 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 14:47:43.825536 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260928 14:47:43.835748 19932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:43.835880 19932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:43.836562 19932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b946d2ee718141879c79571900f56ddd: Sending DeleteTablet for 3 replicas of tablet 463b85605bc6483aac4aa4bf27363850
I20260928 14:47:43.836818 20336 tablet_service.cc:1514] Processing DeleteTablet for tablet 463b85605bc6483aac4aa4bf27363850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:43 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:44870
I20260928 14:47:43.836947 20461 tablet_replica.cc:330] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: stopping tablet replica
I20260928 14:47:43.837015 20461 raft_consensus.cc:2229] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:47:43.837060 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 463b85605bc6483aac4aa4bf27363850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:43 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:42612
I20260928 14:47:43.837059 20071 tablet_service.cc:1514] Processing DeleteTablet for tablet 463b85605bc6483aac4aa4bf27363850 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:43 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:50266
I20260928 14:47:43.837098 20461 raft_consensus.cc:2258] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:43.837143 20463 tablet_replica.cc:330] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: stopping tablet replica
I20260928 14:47:43.837149 20462 tablet_replica.cc:330] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: stopping tablet replica
I20260928 14:47:43.837194 20462 raft_consensus.cc:2229] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:43.837195 20463 raft_consensus.cc:2229] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:43.837230 20462 raft_consensus.cc:2258] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:43.837244 20463 raft_consensus.cc:2258] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:43.837544 20463 ts_tablet_manager.cc:1905] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:43.837558 20462 ts_tablet_manager.cc:1905] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:43.839365 20461 ts_tablet_manager.cc:1905] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:43.839432 20462 ts_tablet_manager.cc:1918] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:47:43.839488 20462 log.cc:1196] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/463b85605bc6483aac4aa4bf27363850
I20260928 14:47:43.839708 20462 ts_tablet_manager.cc:1939] T 463b85605bc6483aac4aa4bf27363850 P 5600caa0d6194387a26a8f7264b80bac: Deleting consensus metadata
I20260928 14:47:43.840128 19918 catalog_manager.cc:4841] TS 5600caa0d6194387a26a8f7264b80bac (127.18.96.130:39633): tablet 463b85605bc6483aac4aa4bf27363850 (table test-table [id=8e5fb3b4e3ef4826a40aee20972b3141]) successfully deleted
I20260928 14:47:43.841380 20461 ts_tablet_manager.cc:1918] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:47:43.841437 20461 log.cc:1196] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/463b85605bc6483aac4aa4bf27363850
I20260928 14:47:43.841658 20461 ts_tablet_manager.cc:1939] T 463b85605bc6483aac4aa4bf27363850 P 22ae489031bc444bb5453bb3f96c2c4f: Deleting consensus metadata
I20260928 14:47:43.842068 19921 catalog_manager.cc:4841] TS 22ae489031bc444bb5453bb3f96c2c4f (127.18.96.131:34279): tablet 463b85605bc6483aac4aa4bf27363850 (table test-table [id=8e5fb3b4e3ef4826a40aee20972b3141]) successfully deleted
I20260928 14:47:43.842785 20463 ts_tablet_manager.cc:1918] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:47:43.842847 20463 log.cc:1196] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/463b85605bc6483aac4aa4bf27363850
I20260928 14:47:43.843057 20463 ts_tablet_manager.cc:1939] T 463b85605bc6483aac4aa4bf27363850 P 9b40a703ae0c4ae7a1a320b5e9bab7ac: Deleting consensus metadata
I20260928 14:47:43.843431 19921 catalog_manager.cc:4841] TS 9b40a703ae0c4ae7a1a320b5e9bab7ac (127.18.96.129:45909): tablet 463b85605bc6483aac4aa4bf27363850 (table test-table [id=8e5fb3b4e3ef4826a40aee20972b3141]) successfully deleted
Sep 28 14:47:43 dist-test-slave-j968 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:47:43.853886 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.843469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.18.96.129:45909)
0928 14:47:43.843524 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.843526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.843854 (+ 328us) server_negotiation.cc:407] Connection header received
0928 14:47:43.843950 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.843951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.843961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.844023 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:43.844722 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.845481 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.846506 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.846808 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.846889 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.850444 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:43.850456 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:43.850457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:43.850465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:43.852465 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.852873 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.852876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.852877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.852944 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.853197 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.853198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.853198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.853378 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:43.853567 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.853633 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.853780 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 14:47:43.854113 20131 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50284
I20260928 14:47:43.862771 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:43.855107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.18.96.190:46547)
0928 14:47:43.855152 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:43.855154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:43.855345 (+ 191us) server_negotiation.cc:407] Connection header received
0928 14:47:43.855415 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:43.855416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:43.855428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:43.855490 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:43.856137 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.856834 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.857869 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:43.858161 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:43.858239 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:43.859780 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:43.859782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:43.859782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:43.859792 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:43.861529 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.861933 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.861934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.861935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.861989 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:43.862181 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:43.862182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:43.862182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:43.862308 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:43.862379 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:43.862586 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 14:47:43.862665 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 14:47:43.863346 19932 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45930
I20260928 14:47:43.864269 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 20006
I20260928 14:47:43.870852 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 20146
I20260928 14:47:43.876020 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 20286
I20260928 14:47:43.880761 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 19902
2026-09-28T14:47:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2519 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:47:43 dist-test-slave-j968 krb5kdc[20501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:47:43 dist-test-slave-j968 krb5kdc[20501](info): set up 2 sockets
Sep 28 14:47:43 dist-test-slave-j968 krb5kdc[20501](info): commencing operation
krb5kdc: starting...
W20260928 14:47:44.916774 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:47:44 dist-test-slave-j968 krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:47:45.035610 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35287
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:35287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:45.116210 20514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:45.116460 20514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:45.116497 20514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:45.117640 20514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:47:45.117713 20514 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:47:45.117736 20514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:45.117759 20514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:45.117789 20514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:47:45.117812 20514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:47:45.119284 20514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:35287
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:45.119560 20514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:45.119792 20514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:47:45.124864 20514 server_base.cc:1034] running on GCE node
W20260928 14:47:45.124838 20519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:45.124848 20522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:45.124935 20520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:45.125370 20514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:45.125617 20514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:45.126785 20514 hybrid_clock.cc:648] HybridClock initialized: now 1790606865126766 us; error 39 us; skew 500 ppm
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:45.129190 20514 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:47:45.129611 20514 webserver.cc:460] Webserver started at http://127.18.96.190:34739/ using document root <none> and password file <none>
I20260928 14:47:45.129834 20514 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:45.129914 20514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:45.130084 20514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:45.131130 20514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "29039138a55b4c35904b7f402a765aab"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "47ef6a7b76126774c989b5054184003f"
server_key_iv: "d6c64c2813e81f1d5b13c634d1dea249"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.131493 20514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "29039138a55b4c35904b7f402a765aab"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "47ef6a7b76126774c989b5054184003f"
server_key_iv: "d6c64c2813e81f1d5b13c634d1dea249"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.132972 20514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:47:45.133734 20529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.133913 20514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.134007 20514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "29039138a55b4c35904b7f402a765aab"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "47ef6a7b76126774c989b5054184003f"
server_key_iv: "d6c64c2813e81f1d5b13c634d1dea249"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.134091 20514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:45.153240 20514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:45.155050 20514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:45.155213 20514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:45.159864 20514 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:35287
I20260928 14:47:45.159946 20591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:35287 every 8 connection(s)
I20260928 14:47:45.160247 20514 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:47:45.160486 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 20514
I20260928 14:47:45.160594 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:47:45.160781 18818 external_mini_cluster.cc:1428] Setting key 6dc540515c384d5ee3a39f2f6bae2a15
I20260928 14:47:45.160981 20592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.164405 20592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab: Bootstrap starting.
I20260928 14:47:45.165026 20592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.165333 20592 log.cc:824] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:45.166054 20592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab: No bootstrap required, opened a new log
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:45.168241 20592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29039138a55b4c35904b7f402a765aab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35287 } }
I20260928 14:47:45.168375 20592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.168398 20592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29039138a55b4c35904b7f402a765aab, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.168540 20592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29039138a55b4c35904b7f402a765aab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35287 } }
I20260928 14:47:45.168627 20592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:47:45.168664 20592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:47:45.168720 20592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:45.169399 20592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29039138a55b4c35904b7f402a765aab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35287 } }
I20260928 14:47:45.169489 20592 leader_election.cc:302] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29039138a55b4c35904b7f402a765aab; no voters:
I20260928 14:47:45.169670 20592 leader_election.cc:288] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:47:45.169701 20597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:45.169809 20597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [term 1 LEADER]: Becoming Leader. State: Replica: 29039138a55b4c35904b7f402a765aab, State: Running, Role: LEADER
I20260928 14:47:45.169890 20597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29039138a55b4c35904b7f402a765aab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35287 } }
I20260928 14:47:45.169943 20592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:47:45.170217 20597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29039138a55b4c35904b7f402a765aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29039138a55b4c35904b7f402a765aab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35287 } } }
I20260928 14:47:45.170361 20597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:45.170567 20597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29039138a55b4c35904b7f402a765aab. Latest consensus state: current_term: 1 leader_uuid: "29039138a55b4c35904b7f402a765aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29039138a55b4c35904b7f402a765aab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35287 } } }
I20260928 14:47:45.170694 20597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:45.171175 20604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:47:45.171746 20604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:47:45.172115 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.161455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.18.96.190:35287)
0928 14:47:45.161655 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.161665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.161700 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:47:45.162173 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.162182 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.162247 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.162403 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:45.162971 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.164351 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.165178 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.165627 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.165745 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.168278 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:45.168282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:45.168282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:45.168292 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:45.170385 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.170833 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.170836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.170836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.170949 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.171149 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.171149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.171150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.171532 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:45.171705 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.171776 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.171938 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260928 14:47:45.173460 20604 catalog_manager.cc:1347] Generated new cluster ID: d10157cf845848ecb29402d71428a9d6
I20260928 14:47:45.173512 20604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:47:45.186328 20604 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:47:45.186880 20604 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:47:45.192310 20604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab: Generated new TSK 0
I20260928 14:47:45.192518 20604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:47:45.223004 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35287
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:45.307973 20618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:45.308177 20618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:45.308203 20618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:45.309305 20618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:45.309340 20618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:45.309360 20618 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:47:45.309371 20618 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:47:45.309388 20618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:47:45.310767 20618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.190:35287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:45.310952 20618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:45.311124 20618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:47:45.316368 20618 server_base.cc:1034] running on GCE node
W20260928 14:47:45.316306 20623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:45.316316 20624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:45.316306 20626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:45.316743 20618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:45.316983 20618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:45.318169 20618 hybrid_clock.cc:648] HybridClock initialized: now 1790606865318150 us; error 29 us; skew 500 ppm
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:45.320735 20618 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:47:45.321224 20618 webserver.cc:460] Webserver started at http://127.18.96.129:39707/ using document root <none> and password file <none>
I20260928 14:47:45.321450 20618 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:45.321517 20618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:45.321677 20618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:45.322778 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "b427479f46c740af8bde76d58b249be4"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "00d123c42d2d5c4c0b738113793a57b6"
server_key_iv: "e348fda698552a4bb315f1ea91f9c379"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.323141 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "b427479f46c740af8bde76d58b249be4"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "00d123c42d2d5c4c0b738113793a57b6"
server_key_iv: "e348fda698552a4bb315f1ea91f9c379"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.324535 20618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:47:45.325271 20633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.325500 20618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:45.325616 20618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "b427479f46c740af8bde76d58b249be4"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "00d123c42d2d5c4c0b738113793a57b6"
server_key_iv: "e348fda698552a4bb315f1ea91f9c379"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.325692 20618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:45.345533 20618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:45.347474 20618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:45.347651 20618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:45.348374 20618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:45.348426 20618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.348456 20618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:45.348479 20618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:45.355587 20618 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:36905
I20260928 14:47:45.355682 20748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:36905 every 8 connection(s)
I20260928 14:47:45.356017 20618 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:47:45.358111 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 20618
I20260928 14:47:45.358152 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.348771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:49919 (local address 127.18.96.190:35287)
0928 14:47:45.348818 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.348820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.349046 (+ 226us) server_negotiation.cc:407] Connection header received
0928 14:47:45.349597 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.349598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.349612 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.349666 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:45.350392 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.351204 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.352523 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.352754 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.352845 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.355249 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:45.355250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:45.355251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:45.355257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:45.356948 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.357321 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.357321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.357322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.357373 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.357546 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.357546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.357547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.357648 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:45.357697 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.357950 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.358044 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:47:45.358222 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:47:45.358258 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.348766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35287 (local address 127.18.96.129:49919)
0928 14:47:45.348910 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.348948 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.349483 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.349744 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.349748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.350087 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:45.350287 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.350292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.351298 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.351300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.352417 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.352420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.352869 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.352870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.353010 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.353566 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:45.353574 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:45.355160 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:45.357022 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.357027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.357028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.357254 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.357415 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.357415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.357415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.357496 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.357704 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:45.357708 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:45.357818 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.357918 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.358038 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260928 14:47:45.358415 18818 external_mini_cluster.cc:1428] Setting key 2afb09ee070776662159ab3953107d9c
I20260928 14:47:45.358788 20749 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35287
I20260928 14:47:45.358875 20749 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:45.359122 20749 heartbeater.cc:511] Master 127.18.96.190:35287 requested a full tablet report, sending...
I20260928 14:47:45.359670 20546 ts_manager.cc:194] Registered new tserver with Master: b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905)
I20260928 14:47:45.360579 20546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:49919
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
I20260928 14:47:45.365999 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.359826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35287 (local address 127.18.96.129:56741)
0928 14:47:45.359978 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.359990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.360043 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.360207 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.360209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.360426 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:45.360507 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.360509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.361423 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.361424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.361974 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.361976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.362368 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.362369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.362444 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.362911 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:45.362917 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:45.363209 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:45.364939 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.364940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.364941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.365116 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.365334 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.365335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.365336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.365397 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.365658 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:45.365659 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:45.365705 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.365795 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.365899 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260928 14:47:45.365999 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.359994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:56741 (local address 127.18.96.190:35287)
0928 14:47:45.360029 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.360032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.360040 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:47:45.360091 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.360092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.360103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.360155 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:45.360609 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.361307 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.362065 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.362279 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.362365 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.363318 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:45.363319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:45.363320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:45.363326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:45.364833 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.365202 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.365203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.365204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.365260 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.365475 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.365476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.365476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.365591 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:45.365666 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.365819 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.365897 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:47:45.382256 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35287
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:45.463141 20758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:45.463435 20758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:45.463480 20758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:45.464707 20758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:45.464779 20758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:45.464815 20758 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:47:45.464845 20758 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:47:45.464885 20758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:47:45.466364 20758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.190:35287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:45.466639 20758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:45.466900 20758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:45.471990 20764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:45.471999 20763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:45.472193 20758 server_base.cc:1034] running on GCE node
W20260928 14:47:45.472018 20766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:45.472415 20758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:45.472673 20758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:45.473845 20758 hybrid_clock.cc:648] HybridClock initialized: now 1790606865473830 us; error 24 us; skew 500 ppm
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:45.476472 20758 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:47:45.476922 20758 webserver.cc:460] Webserver started at http://127.18.96.130:34247/ using document root <none> and password file <none>
I20260928 14:47:45.477154 20758 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:45.477222 20758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:45.477381 20758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:45.478435 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "77da14f948644a9f83f1154ed8f74fdb"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "ab6755fa0fdea2d89eb78926d9245781"
server_key_iv: "2fc634e53e0ed4cfba99c3b120c259d3"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.478811 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "77da14f948644a9f83f1154ed8f74fdb"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "ab6755fa0fdea2d89eb78926d9245781"
server_key_iv: "2fc634e53e0ed4cfba99c3b120c259d3"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.480188 20758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:47:45.480916 20773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.481084 20758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:47:45.481187 20758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "77da14f948644a9f83f1154ed8f74fdb"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "ab6755fa0fdea2d89eb78926d9245781"
server_key_iv: "2fc634e53e0ed4cfba99c3b120c259d3"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.481261 20758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:45.488086 20758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:45.490331 20758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:45.490501 20758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:45.490991 20758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:45.491052 20758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.491129 20758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:45.491184 20758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.498148 20758 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:37117
I20260928 14:47:45.498224 20888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:37117 every 8 connection(s)
I20260928 14:47:45.498533 20758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:45.501734 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.491688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:46205 (local address 127.18.96.190:35287)
0928 14:47:45.491739 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.491740 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.492035 (+ 295us) server_negotiation.cc:407] Connection header received
0928 14:47:45.492619 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.492621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.492637 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.492715 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:45.493720 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.494591 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.496178 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.496502 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.496582 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.499170 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:45.499171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:45.499171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:45.499178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:45.500592 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.500947 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.500948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.500948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.500997 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.501206 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.501207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.501207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.501309 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:45.501382 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.501602 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.501669 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:47:45.501914 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.491667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35287 (local address 127.18.96.130:46205)
0928 14:47:45.491850 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.491938 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.492460 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.492890 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.492895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.493289 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:45.493601 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.493608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.494700 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.494702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.496083 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.496088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.496601 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.496602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.496750 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.497376 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:45.497388 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:45.499080 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:45.500663 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.500669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.500671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.500882 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.501056 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.501056 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.501057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.501153 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.501367 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:45.501371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:45.501485 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.501565 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.501682 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260928 14:47:45.502405 20889 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35287
I20260928 14:47:45.502517 20889 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:45.502689 20889 heartbeater.cc:511] Master 127.18.96.190:35287 requested a full tablet report, sending...
I20260928 14:47:45.503106 20546 ts_manager.cc:194] Registered new tserver with Master: 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130:37117)
I20260928 14:47:45.503520 20546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:46205
I20260928 14:47:45.506970 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 20758
I20260928 14:47:45.507046 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:47:45.507236 18818 external_mini_cluster.cc:1428] Setting key 814d7fd025f488f2b49da30cf30e7dab
I20260928 14:47:45.509086 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.503546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:54509 (local address 127.18.96.190:35287)
0928 14:47:45.503595 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.503597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.503651 (+ 54us) server_negotiation.cc:407] Connection header received
0928 14:47:45.503721 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.503722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.503732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.503775 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:45.504149 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.504608 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.505304 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.505504 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.505584 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.506466 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:45.506468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:45.506468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:45.506475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:45.507908 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.508279 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.508280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.508280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.508340 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.508535 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.508536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.508537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.508693 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:45.508767 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.508924 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.508997 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:47:45.509104 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.503461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35287 (local address 127.18.96.130:54509)
0928 14:47:45.503591 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.503603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.503665 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.503801 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.503801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.504020 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:45.504071 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.504072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.504697 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.504698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.505210 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.505212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.505585 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.505586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.505658 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.506088 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:45.506093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:45.506359 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:45.508013 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.508014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.508015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.508193 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.508406 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.508407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.508407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.508471 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.508770 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:45.508771 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:45.508815 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.508900 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.508999 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:47:45.532461 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35287
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:45.616878 20898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:45.617165 20898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:45.617209 20898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:45.618505 20898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:45.618583 20898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:45.618623 20898 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:47:45.618649 20898 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:47:45.618686 20898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:47:45.620234 20898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.190:35287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:45.620498 20898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:45.620733 20898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:45.625854 20906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:45.625850 20904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:45.625850 20903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:45.626230 20898 server_base.cc:1034] running on GCE node
I20260928 14:47:45.626461 20898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:45.626704 20898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:45.627907 20898 hybrid_clock.cc:648] HybridClock initialized: now 1790606865627845 us; error 37 us; skew 500 ppm
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:45.630398 20898 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:47:45.630894 20898 webserver.cc:460] Webserver started at http://127.18.96.131:33821/ using document root <none> and password file <none>
I20260928 14:47:45.631150 20898 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:45.631196 20898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:45.631338 20898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:45.632474 20898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "3afaba5fae404ff6a181dda9104b7ba4"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "bc464f96c2df3fdef93c7a941a39821c"
server_key_iv: "5f04fbef98f9c4a2d4beb936de0f9372"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.632838 20898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "3afaba5fae404ff6a181dda9104b7ba4"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "bc464f96c2df3fdef93c7a941a39821c"
server_key_iv: "5f04fbef98f9c4a2d4beb936de0f9372"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.634289 20898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:47:45.635102 20913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.635291 20898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:45.635393 20898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "3afaba5fae404ff6a181dda9104b7ba4"
format_stamp: "Formatted at 2026-09-28 14:47:45 on dist-test-slave-j968"
server_key: "bc464f96c2df3fdef93c7a941a39821c"
server_key_iv: "5f04fbef98f9c4a2d4beb936de0f9372"
server_key_version: "encryptionkey@0"
I20260928 14:47:45.635468 20898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:45.658692 20898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:45.660969 20898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:45.661144 20898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:45.661720 20898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:45.661779 20898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.661837 20898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:45.661883 20898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:45.669152 20898 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:43835
I20260928 14:47:45.669227 21028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:43835 every 8 connection(s)
I20260928 14:47:45.669564 20898 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:47:45.672628 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.662129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:60317 (local address 127.18.96.190:35287)
0928 14:47:45.662199 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.662202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.662464 (+ 262us) server_negotiation.cc:407] Connection header received
0928 14:47:45.662998 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.662999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.663012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.663067 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:45.663761 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.664412 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.665440 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.665679 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.665767 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.669344 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:45.669345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:45.669346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:45.669354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:45.671077 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.671574 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.671575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.671576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.671641 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.671963 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.671964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.671964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.672091 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:45.672171 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.672450 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.672536 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:47:45.672832 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.662136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35287 (local address 127.18.96.131:60317)
0928 14:47:45.662302 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.662342 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.662889 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.663149 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.663153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.663460 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:45.663661 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.663666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.664515 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.664516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.665324 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.665326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.665776 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.665777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.665953 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.666480 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:45.666492 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:45.669244 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:45.671170 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.671176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.671179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.671503 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.671735 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.671736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.671736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.671854 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.672159 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:45.672165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:45.672311 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.672426 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.672591 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260928 14:47:45.673311 21029 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35287
I20260928 14:47:45.673391 21029 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:45.673595 21029 heartbeater.cc:511] Master 127.18.96.190:35287 requested a full tablet report, sending...
I20260928 14:47:45.673961 20545 ts_manager.cc:194] Registered new tserver with Master: 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:45.674341 20545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:60317
I20260928 14:47:45.677577 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 20898
I20260928 14:47:45.677675 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:47:45.677940 18818 external_mini_cluster.cc:1428] Setting key 966c65bce8f515f4d31650be3013a836
I20260928 14:47:45.679294 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:47:45.680822 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.674423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:55807 (local address 127.18.96.190:35287)
0928 14:47:45.674468 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.674470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.674585 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:47:45.674658 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.674659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.674671 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.674719 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:45.675008 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.675512 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.676439 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.676625 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.676681 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.677882 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:45.677883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:45.677883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:45.677889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:45.679636 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.680161 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.680162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.680162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.680212 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.680412 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.680412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.680412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.680504 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:45.680550 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.680706 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.680763 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:47:45.680989 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.674362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35287 (local address 127.18.96.131:55807)
0928 14:47:45.674516 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.674527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.674588 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.674727 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.674728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.674876 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:45.674933 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.674934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.675623 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.675624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.676340 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.676342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.676696 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.676697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.676761 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.677307 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:45.677313 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:45.677798 (+ 485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:45.679728 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.679729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.679730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.680088 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.680281 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.680282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.680283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.680355 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.680563 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:45.680564 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:45.680615 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.680690 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.680809 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:45.703550 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.696570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.18.96.190:35287)
0928 14:47:45.696635 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.696637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.696646 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:47:45.696712 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.696713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.696721 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.696772 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:45.697317 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.697825 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.698536 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.698754 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.698823 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.701092 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:45.701093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:45.701094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:45.701102 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:45.702549 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.702863 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.702863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.702864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.702913 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.703041 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.703041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.703041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.703141 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:45.703213 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.703364 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.703424 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:47:45.705060 20545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:47:45.706570 20545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:45.713371 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.709236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36905 (local address 127.0.0.1:60344)
0928 14:47:45.709355 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.709365 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.709406 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.709886 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.709887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.709889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:45.709946 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.709948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.710884 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.710885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.712205 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.712208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.713028 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.713029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.713117 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.713131 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.713228 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.713287 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260928 14:47:45.713445 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.709085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:43835 (local address 127.0.0.1:41956)
0928 14:47:45.709299 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.709309 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.709362 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.710094 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.710095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.710098 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:45.710148 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.710149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.711314 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.711315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.712277 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.712279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.713196 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.713196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.713265 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.713268 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.713317 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.713349 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260928 14:47:45.713657 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.709145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.18.96.130:37117)
0928 14:47:45.709634 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.709638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.709646 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:47:45.709677 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.709678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.709690 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.709820 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:45.709994 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.711160 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.712411 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.713138 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.713199 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.713249 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.713490 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.713557 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":72,"threads_started":1}
I20260928 14:47:45.713682 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.709463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.18.96.129:36905)
0928 14:47:45.709626 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.709630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.709638 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:47:45.709669 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.709671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.709683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.709773 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:45.710028 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.710793 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.712294 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.712931 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.713024 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.713088 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.713575 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.713611 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260928 14:47:45.713824 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.709034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:37117 (local address 127.0.0.1:48888)
0928 14:47:45.709235 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.709281 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.709349 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.709828 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.709831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.709833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:45.709905 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.709907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.711251 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.711252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.712323 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.712325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.713286 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.713287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.713350 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.713354 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.713403 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.713445 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":59,"threads_started":1}
I20260928 14:47:45.713994 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.709173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.18.96.131:43835)
0928 14:47:45.709945 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.709949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.709958 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:47:45.709994 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.709995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.710010 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.710083 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:45.710225 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.711256 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.712369 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.713109 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.713192 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.713249 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.713328 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.713365 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":80,"threads_started":1}
I20260928 14:47:45.714423 20683 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfdf2aca10d4d2d8abe35e025c5ed19 (DEFAULT_TABLE table=test-table [id=daa617b956e44a1eaf173798933764a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:45.714468 20823 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfdf2aca10d4d2d8abe35e025c5ed19 (DEFAULT_TABLE table=test-table [id=daa617b956e44a1eaf173798933764a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:45.714814 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfdf2aca10d4d2d8abe35e025c5ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.714815 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfdf2aca10d4d2d8abe35e025c5ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.714725 20963 tablet_service.cc:1467] Processing CreateTablet for tablet 1bfdf2aca10d4d2d8abe35e025c5ed19 (DEFAULT_TABLE table=test-table [id=daa617b956e44a1eaf173798933764a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:45.715022 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bfdf2aca10d4d2d8abe35e025c5ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.717769 21052 tablet_bootstrap.cc:492] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: Bootstrap starting.
I20260928 14:47:45.718147 21053 tablet_bootstrap.cc:492] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: Bootstrap starting.
I20260928 14:47:45.718505 21052 tablet_bootstrap.cc:654] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.718700 21054 tablet_bootstrap.cc:492] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: Bootstrap starting.
I20260928 14:47:45.718917 21052 log.cc:824] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:45.719054 21053 tablet_bootstrap.cc:654] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.719310 21054 tablet_bootstrap.cc:654] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.719398 21053 log.cc:824] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:45.719645 21054 log.cc:824] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:45.719694 21052 tablet_bootstrap.cc:492] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: No bootstrap required, opened a new log
I20260928 14:47:45.719740 21052 ts_tablet_manager.cc:1397] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:47:45.720194 21053 tablet_bootstrap.cc:492] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: No bootstrap required, opened a new log
I20260928 14:47:45.720268 21053 ts_tablet_manager.cc:1397] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:47:45.720391 21054 tablet_bootstrap.cc:492] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: No bootstrap required, opened a new log
I20260928 14:47:45.720479 21054 ts_tablet_manager.cc:1397] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:47:45.721822 21052 raft_consensus.cc:348] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.722019 21052 raft_consensus.cc:374] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.722074 21052 raft_consensus.cc:729] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da14f948644a9f83f1154ed8f74fdb, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.722199 21052 consensus_queue.cc:260] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.722126 21054 raft_consensus.cc:348] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.722270 21054 raft_consensus.cc:374] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.722304 21054 raft_consensus.cc:729] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b427479f46c740af8bde76d58b249be4, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.722406 21054 consensus_queue.cc:260] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.722483 21052 ts_tablet_manager.cc:1428] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:47:45.722395 21053 raft_consensus.cc:348] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.722527 20889 heartbeater.cc:503] Master 127.18.96.190:35287 was elected leader, sending a full tablet report...
I20260928 14:47:45.722532 21053 raft_consensus.cc:374] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.722584 21053 raft_consensus.cc:729] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afaba5fae404ff6a181dda9104b7ba4, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.722657 21054 ts_tablet_manager.cc:1428] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:47:45.722731 21053 consensus_queue.cc:260] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.722805 20749 heartbeater.cc:503] Master 127.18.96.190:35287 was elected leader, sending a full tablet report...
I20260928 14:47:45.722977 21053 ts_tablet_manager.cc:1428] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:47:45.723157 21029 heartbeater.cc:503] Master 127.18.96.190:35287 was elected leader, sending a full tablet report...
W20260928 14:47:45.749118 20890 tablet.cc:2369] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:47:45.780617 21058 raft_consensus.cc:482] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:45.780802 21058 raft_consensus.cc:504] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.781245 21058 leader_election.cc:288] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:45.784186 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.781530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:50813 (local address 127.18.96.129:36905)
0928 14:47:45.781585 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.781587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.781598 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:47:45.781672 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.781673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.781683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.781742 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:45.781945 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.782412 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.783357 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.783787 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.783931 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.783941 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.784085 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.784119 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:47:45.784185 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.781479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36905 (local address 127.18.96.130:50813)
0928 14:47:45.781549 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.781562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.781611 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.781766 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.781777 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.781779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:45.781849 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.781851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.782498 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.782499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.783290 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.783292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.783911 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.783912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.783994 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.783997 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.784080 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.784119 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:47:45.784598 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.781486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:43835 (local address 127.18.96.130:39451)
0928 14:47:45.781762 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.781774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.781839 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.781983 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.781984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.781985 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:45.782059 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.782061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.782685 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.782686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.783613 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.783615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.784374 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.784376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.784456 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.784465 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.784509 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.784536 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":53,"thread_start_us":135,"threads_started":1}
I20260928 14:47:45.784664 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.781707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:39451 (local address 127.18.96.131:43835)
0928 14:47:45.781770 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.781774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.781822 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:47:45.781906 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.781907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.781916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.781971 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:45.782136 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.782602 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.783686 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.784218 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.784311 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.784320 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.784546 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.784594 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:47:45.784615 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfdf2aca10d4d2d8abe35e025c5ed19" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b427479f46c740af8bde76d58b249be4" is_pre_election: true
I20260928 14:47:45.784785 20703 raft_consensus.cc:2454] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 0.
I20260928 14:47:45.785066 20776 leader_election.cc:302] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da14f948644a9f83f1154ed8f74fdb, b427479f46c740af8bde76d58b249be4; no voters:
I20260928 14:47:45.785210 21058 raft_consensus.cc:2781] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:45.785274 21058 raft_consensus.cc:482] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:45.785310 21058 raft_consensus.cc:3037] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:45.785266 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfdf2aca10d4d2d8abe35e025c5ed19" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4" is_pre_election: true
I20260928 14:47:45.785419 20983 raft_consensus.cc:2454] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 0.
I20260928 14:47:45.786373 21058 raft_consensus.cc:504] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.786522 21058 leader_election.cc:288] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 election: Requested vote from peers b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:45.786652 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfdf2aca10d4d2d8abe35e025c5ed19" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4"
I20260928 14:47:45.786679 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bfdf2aca10d4d2d8abe35e025c5ed19" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b427479f46c740af8bde76d58b249be4"
I20260928 14:47:45.786751 20983 raft_consensus.cc:3037] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:45.786748 20703 raft_consensus.cc:3037] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:45.787415 20983 raft_consensus.cc:2454] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 1.
I20260928 14:47:45.787475 20703 raft_consensus.cc:2454] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 1.
I20260928 14:47:45.787649 20776 leader_election.cc:302] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da14f948644a9f83f1154ed8f74fdb, b427479f46c740af8bde76d58b249be4; no voters:
I20260928 14:47:45.787756 21058 raft_consensus.cc:2781] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:45.787947 21058 raft_consensus.cc:686] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 1 LEADER]: Becoming Leader. State: Replica: 77da14f948644a9f83f1154ed8f74fdb, State: Running, Role: LEADER
I20260928 14:47:45.788058 21058 consensus_queue.cc:237] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.788848 20546 catalog_manager.cc:5495] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130). New cstate: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:47:45 dist-test-slave-j968 krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606865, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:45.836035 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.828726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.18.96.190:35287)
0928 14:47:45.828812 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.828815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.828937 (+ 122us) server_negotiation.cc:407] Connection header received
0928 14:47:45.829025 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.829026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.829035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.829104 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:45.829678 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.830167 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.830900 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.831085 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.831168 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.833455 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:45.833456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:45.833457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:45.833464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:45.834819 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.835201 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.835202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.835203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.835256 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:45.835455 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:45.835456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:45.835457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:45.835564 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:45.835643 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.835801 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.835921 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260928 14:47:45.836143 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.828651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35287 (local address 127.0.0.1:33890)
0928 14:47:45.828851 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.828861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.828956 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.829108 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.829109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.829548 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:45.829597 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.829599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.830248 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.830249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.830797 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.830799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.831170 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.831171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.831227 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.831934 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:45.831939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:45.833364 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:45.834903 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.834911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.834914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.835112 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.835331 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:45.835332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:45.835332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:45.835395 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:45.835643 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:45.835651 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:45.835695 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.835779 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.835983 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":96,"thread_start_us":123,"threads_started":1}
I20260928 14:47:45.838125 20546 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:33890:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:47:45.838402 20546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:45.840019 20823 tablet_service.cc:1467] Processing CreateTablet for tablet a8035a5ee1334bbdbf921d5535f8e9a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96f7397a1f274c37a0fd4cf8d02d0d4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:47:45.840008 20683 tablet_service.cc:1467] Processing CreateTablet for tablet a8035a5ee1334bbdbf921d5535f8e9a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96f7397a1f274c37a0fd4cf8d02d0d4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:47:45.840186 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8035a5ee1334bbdbf921d5535f8e9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.840190 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8035a5ee1334bbdbf921d5535f8e9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.841360 21054 tablet_bootstrap.cc:492] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: Bootstrap starting.
I20260928 14:47:45.841418 21052 tablet_bootstrap.cc:492] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: Bootstrap starting.
I20260928 14:47:45.841475 20963 tablet_service.cc:1467] Processing CreateTablet for tablet a8035a5ee1334bbdbf921d5535f8e9a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=96f7397a1f274c37a0fd4cf8d02d0d4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:47:45.841632 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8035a5ee1334bbdbf921d5535f8e9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.841792 21052 tablet_bootstrap.cc:654] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.842087 21054 tablet_bootstrap.cc:654] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.842468 21052 tablet_bootstrap.cc:492] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: No bootstrap required, opened a new log
I20260928 14:47:45.842525 21052 ts_tablet_manager.cc:1397] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:45.842682 21052 raft_consensus.cc:348] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.842738 21054 tablet_bootstrap.cc:492] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: No bootstrap required, opened a new log
I20260928 14:47:45.842772 21052 raft_consensus.cc:374] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.842783 21054 ts_tablet_manager.cc:1397] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:45.842798 21052 raft_consensus.cc:729] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da14f948644a9f83f1154ed8f74fdb, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.842849 21052 consensus_queue.cc:260] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.842948 21052 ts_tablet_manager.cc:1428] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 14:47:45.842924 21054 raft_consensus.cc:348] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.843011 21054 raft_consensus.cc:374] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.843034 21054 raft_consensus.cc:729] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b427479f46c740af8bde76d58b249be4, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.842984 21058 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.843063 21058 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:45.843096 21054 consensus_queue.cc:260] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.843187 21054 ts_tablet_manager.cc:1428] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.843261 21053 tablet_bootstrap.cc:492] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: Bootstrap starting.
I20260928 14:47:45.843271 21059 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.843351 21059 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:45.843830 21053 tablet_bootstrap.cc:654] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.844524 21053 tablet_bootstrap.cc:492] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: No bootstrap required, opened a new log
I20260928 14:47:45.844565 21053 ts_tablet_manager.cc:1397] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:45.844672 21053 raft_consensus.cc:348] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.844720 21053 raft_consensus.cc:374] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.844733 21053 raft_consensus.cc:729] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afaba5fae404ff6a181dda9104b7ba4, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.844801 21053 consensus_queue.cc:260] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.844890 21053 ts_tablet_manager.cc:1428] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 14:47:45.844947 21060 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.845039 21060 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 14:47:45.856855 20750 tablet.cc:2369] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:47:45.862278 21058 raft_consensus.cc:482] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:45.862381 21058 raft_consensus.cc:504] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.862566 21058 leader_election.cc:288] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:45.862787 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8035a5ee1334bbdbf921d5535f8e9a1" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b427479f46c740af8bde76d58b249be4" is_pre_election: true
I20260928 14:47:45.862779 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8035a5ee1334bbdbf921d5535f8e9a1" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4" is_pre_election: true
I20260928 14:47:45.862887 20703 raft_consensus.cc:2454] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 0.
I20260928 14:47:45.862887 20983 raft_consensus.cc:2454] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 0.
I20260928 14:47:45.863044 20774 leader_election.cc:302] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3afaba5fae404ff6a181dda9104b7ba4, 77da14f948644a9f83f1154ed8f74fdb; no voters:
I20260928 14:47:45.863147 21058 raft_consensus.cc:2781] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:45.863217 21058 raft_consensus.cc:482] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:45.863256 21058 raft_consensus.cc:3037] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:45.864013 21058 raft_consensus.cc:504] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.864159 21058 leader_election.cc:288] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 election: Requested vote from peers b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:45.864339 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8035a5ee1334bbdbf921d5535f8e9a1" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4"
I20260928 14:47:45.864322 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8035a5ee1334bbdbf921d5535f8e9a1" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b427479f46c740af8bde76d58b249be4"
I20260928 14:47:45.864405 20983 raft_consensus.cc:3037] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:45.864405 20703 raft_consensus.cc:3037] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:45.865161 20703 raft_consensus.cc:2454] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 1.
I20260928 14:47:45.865161 20983 raft_consensus.cc:2454] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 1.
I20260928 14:47:45.865329 20774 leader_election.cc:302] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3afaba5fae404ff6a181dda9104b7ba4, 77da14f948644a9f83f1154ed8f74fdb; no voters:
I20260928 14:47:45.865429 21058 raft_consensus.cc:2781] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:45.865482 21058 raft_consensus.cc:686] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [term 1 LEADER]: Becoming Leader. State: Replica: 77da14f948644a9f83f1154ed8f74fdb, State: Running, Role: LEADER
I20260928 14:47:45.865535 21058 consensus_queue.cc:237] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.865881 21062 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: TxnStatusTablet state changed. Reason: New leader 77da14f948644a9f83f1154ed8f74fdb. Latest consensus state: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.866000 21062 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:47:45.866160 20545 catalog_manager.cc:5495] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130). New cstate: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } health_report { overall_health: UNKNOWN } } }
I20260928 14:47:45.866240 21074 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:47:45.866376 21074 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:47:45.873409 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.870763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:37117 (local address 127.0.0.1:48894)
0928 14:47:45.870841 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.870853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.870905 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.871135 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.871136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.871137 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:47:45.871190 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.871192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.871845 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.871846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.872449 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.872452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.872836 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.872837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.872907 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.872946 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:47:45.873231 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:47:45.873239 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.873284 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.873333 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 14:47:45.873428 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.870797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.18.96.130:37117)
0928 14:47:45.870835 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.870837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.870888 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:47:45.871019 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.871020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.871037 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.871127 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:45.871268 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.871761 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.872538 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.872749 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.872834 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.873061 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:45.873158 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:45.873239 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.873288 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.873311 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:47:45.875089 20983 raft_consensus.cc:1261] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Refusing update from remote peer 77da14f948644a9f83f1154ed8f74fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:45.875123 20703 raft_consensus.cc:1261] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Refusing update from remote peer 77da14f948644a9f83f1154ed8f74fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:45.875468 21058 consensus_queue.cc:1035] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:45.875655 21062 consensus_queue.cc:1035] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:45.876699 21059 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: TxnStatusTablet state changed. Reason: New leader 77da14f948644a9f83f1154ed8f74fdb. Latest consensus state: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.876710 21060 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: TxnStatusTablet state changed. Reason: New leader 77da14f948644a9f83f1154ed8f74fdb. Latest consensus state: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.876798 21059 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:45.876818 21060 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:45.878147 21058 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.878239 21058 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:47:45.878336 21058 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.878383 21058 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P 77da14f948644a9f83f1154ed8f74fdb: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:47:45.878417 21059 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.878492 21059 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P b427479f46c740af8bde76d58b249be4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:45.878758 21079 mvcc.cc:205] Tried to move back new op lower bound from 7334325722621820928 to 7334325722585608192. Current Snapshot: MvccSnapshot[applied={T|T < 7334325722621820928}]
I20260928 14:47:45.878891 21060 tablet_replica.cc:439] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } } }
I20260928 14:47:45.878962 21060 tablet_replica.cc:442] T a8035a5ee1334bbdbf921d5535f8e9a1 P 3afaba5fae404ff6a181dda9104b7ba4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:47:45.879802 21078 mvcc.cc:205] Tried to move back new op lower bound from 7334325722621820928 to 7334325722585608192. Current Snapshot: MvccSnapshot[applied={T|T < 7334325722621820928 or (T in {7334325722621820928})}]
I20260928 14:47:45.882923 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.880661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.18.96.130:37117)
0928 14:47:45.880725 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.880727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.880735 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:47:45.880759 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.880760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.880769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.880824 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:45.880964 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.881431 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.882207 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.882411 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.882501 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.882662 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:45.882700 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:45.882753 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.882821 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.882844 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:47:45.887761 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.885269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:37117 (local address 127.18.96.130:36645)
0928 14:47:45.885341 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:45.885352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:45.885389 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:45.885516 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:45.885516 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:45.885517 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:47:45.885566 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.885567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.886376 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.886377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.886926 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:45.886928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.887375 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.887376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:45.887421 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.887434 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:47:45.887617 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:47:45.887618 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:47:45.887652 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 14:47:45.887684 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":2}
I20260928 14:47:45.887770 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:45.885329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:36645 (local address 127.18.96.130:37117)
0928 14:47:45.885365 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:45.885367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:45.885377 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:47:45.885441 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:45.885442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:45.885450 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:45.885515 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:45.885602 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.886300 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.887006 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:45.887303 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:45.887372 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:45.887520 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:45.887552 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:45.887613 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:45.887655 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 14:47:45.887678 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
I20260928 14:47:45.890101 20703 raft_consensus.cc:1261] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Refusing update from remote peer 77da14f948644a9f83f1154ed8f74fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:45.890100 20983 raft_consensus.cc:1261] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Refusing update from remote peer 77da14f948644a9f83f1154ed8f74fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:45.890301 21058 consensus_queue.cc:1035] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:45.890400 21062 consensus_queue.cc:1035] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:45.892357 21077 mvcc.cc:205] Tried to move back new op lower bound from 7334325722684538880 to 7334325722268368896. Current Snapshot: MvccSnapshot[applied={T|T < 7334325722684538880}]
I20260928 14:47:45.904850 20545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:45.904950 20545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:45.905763 20545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab: Sending DeleteTablet for 3 replicas of tablet 1bfdf2aca10d4d2d8abe35e025c5ed19
I20260928 14:47:45.906113 20823 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bfdf2aca10d4d2d8abe35e025c5ed19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:45 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:48888
I20260928 14:47:45.906165 20683 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bfdf2aca10d4d2d8abe35e025c5ed19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:45 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:60344
I20260928 14:47:45.906172 20963 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bfdf2aca10d4d2d8abe35e025c5ed19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:45 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:41956
I20260928 14:47:45.906558 21094 tablet_replica.cc:330] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: stopping tablet replica
I20260928 14:47:45.906715 21094 raft_consensus.cc:2229] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:45.906652 20545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:47:45.906803 20545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:45.906883 21094 raft_consensus.cc:2258] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:45.907814 21094 ts_tablet_manager.cc:1905] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:45.907855 20683 tablet_service.cc:1467] Processing CreateTablet for tablet f72cf236cdc94f97a96b5fcd7acd61e8 (DEFAULT_TABLE table=test-table [id=ced25a20bb344771a27e432e4d3af9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:45.907929 20963 tablet_service.cc:1467] Processing CreateTablet for tablet f72cf236cdc94f97a96b5fcd7acd61e8 (DEFAULT_TABLE table=test-table [id=ced25a20bb344771a27e432e4d3af9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:45.908025 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f72cf236cdc94f97a96b5fcd7acd61e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.908026 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f72cf236cdc94f97a96b5fcd7acd61e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.908128 21092 tablet_replica.cc:330] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: stopping tablet replica
I20260928 14:47:45.908226 21092 raft_consensus.cc:2229] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:47:45.908313 21092 raft_consensus.cc:2258] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:45.909329 21053 tablet_bootstrap.cc:492] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: Bootstrap starting.
I20260928 14:47:45.909329 21054 tablet_bootstrap.cc:492] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: Bootstrap starting.
I20260928 14:47:45.909751 21053 tablet_bootstrap.cc:654] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.909751 21054 tablet_bootstrap.cc:654] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.910228 21092 ts_tablet_manager.cc:1905] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:45.910473 21053 tablet_bootstrap.cc:492] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: No bootstrap required, opened a new log
I20260928 14:47:45.910487 20823 tablet_service.cc:1467] Processing CreateTablet for tablet f72cf236cdc94f97a96b5fcd7acd61e8 (DEFAULT_TABLE table=test-table [id=ced25a20bb344771a27e432e4d3af9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:45.910524 21053 ts_tablet_manager.cc:1397] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:45.910586 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f72cf236cdc94f97a96b5fcd7acd61e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:45.910660 21053 raft_consensus.cc:348] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.910733 21053 raft_consensus.cc:374] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.910753 21053 raft_consensus.cc:729] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afaba5fae404ff6a181dda9104b7ba4, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.910795 21053 consensus_queue.cc:260] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.910871 21053 ts_tablet_manager.cc:1428] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.911846 21092 ts_tablet_manager.cc:1918] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:47:45.911949 21092 log.cc:1196] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/1bfdf2aca10d4d2d8abe35e025c5ed19
I20260928 14:47:45.911965 21052 tablet_bootstrap.cc:492] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: Bootstrap starting.
I20260928 14:47:45.912204 21092 ts_tablet_manager.cc:1939] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 77da14f948644a9f83f1154ed8f74fdb: Deleting consensus metadata
I20260928 14:47:45.912320 21052 tablet_bootstrap.cc:654] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:45.912372 21094 ts_tablet_manager.cc:1918] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:47:45.912432 21094 log.cc:1196] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/1bfdf2aca10d4d2d8abe35e025c5ed19
I20260928 14:47:45.912686 21094 ts_tablet_manager.cc:1939] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P 3afaba5fae404ff6a181dda9104b7ba4: Deleting consensus metadata
I20260928 14:47:45.912693 20532 catalog_manager.cc:4841] TS 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130:37117): tablet 1bfdf2aca10d4d2d8abe35e025c5ed19 (table test-table [id=daa617b956e44a1eaf173798933764a1]) successfully deleted
I20260928 14:47:45.913035 21093 tablet_replica.cc:330] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: stopping tablet replica
I20260928 14:47:45.913131 20530 catalog_manager.cc:4841] TS 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835): tablet 1bfdf2aca10d4d2d8abe35e025c5ed19 (table test-table [id=daa617b956e44a1eaf173798933764a1]) successfully deleted
I20260928 14:47:45.913174 21093 raft_consensus.cc:2229] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:45.913189 21054 tablet_bootstrap.cc:492] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: No bootstrap required, opened a new log
I20260928 14:47:45.913239 21054 ts_tablet_manager.cc:1397] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 14:47:45.913443 21052 tablet_bootstrap.cc:492] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: No bootstrap required, opened a new log
I20260928 14:47:45.913396 21054 raft_consensus.cc:348] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.913477 21054 raft_consensus.cc:374] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.913480 21052 ts_tablet_manager.cc:1397] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:47:45.913501 21054 raft_consensus.cc:729] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b427479f46c740af8bde76d58b249be4, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.913554 21054 consensus_queue.cc:260] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.913645 21054 ts_tablet_manager.cc:1428] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:45.913630 21052 raft_consensus.cc:348] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.913692 21052 raft_consensus.cc:374] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:45.913715 21052 raft_consensus.cc:729] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da14f948644a9f83f1154ed8f74fdb, State: Initialized, Role: FOLLOWER
I20260928 14:47:45.913759 21052 consensus_queue.cc:260] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:45.913849 21052 ts_tablet_manager.cc:1428] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 14:47:45.913242 21093 raft_consensus.cc:2258] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:45.916424 21093 ts_tablet_manager.cc:1905] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:45.917979 21093 ts_tablet_manager.cc:1918] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:47:45.918057 21093 log.cc:1196] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/1bfdf2aca10d4d2d8abe35e025c5ed19
I20260928 14:47:45.918310 21093 ts_tablet_manager.cc:1939] T 1bfdf2aca10d4d2d8abe35e025c5ed19 P b427479f46c740af8bde76d58b249be4: Deleting consensus metadata
I20260928 14:47:45.918728 20532 catalog_manager.cc:4841] TS b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905): tablet 1bfdf2aca10d4d2d8abe35e025c5ed19 (table test-table [id=daa617b956e44a1eaf173798933764a1]) successfully deleted
W20260928 14:47:45.920521 21030 tablet.cc:2369] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:47:46.016073 21059 raft_consensus.cc:482] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:46.016177 21059 raft_consensus.cc:504] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:46.016597 21059 leader_election.cc:288] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130:37117), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:46.020139 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:46.016827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:54431 (local address 127.18.96.130:37117)
0928 14:47:46.016931 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:46.016933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:46.016946 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:47:46.017023 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:46.017024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:46.017035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:46.017085 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:46.017299 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.018040 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.019387 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.019754 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.019844 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:46.019854 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:46.020029 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 14:47:46.020068 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:47:46.020165 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:46.016754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:37117 (local address 127.18.96.129:54431)
0928 14:47:46.016883 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:46.016895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:46.016955 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:46.017138 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:46.017140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:46.017142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:46.017220 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:46.017222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.018115 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.018116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:46.019258 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:46.019260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.019841 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.019842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:46.019947 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:46.019951 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:47:46.020022 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:47:46.020065 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":5,"thread_start_us":63,"threads_started":1}
I20260928 14:47:46.020383 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:46.017001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:36077 (local address 127.18.96.131:43835)
0928 14:47:46.017042 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:46.017045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:46.017120 (+ 75us) server_negotiation.cc:407] Connection header received
0928 14:47:46.017187 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:46.017188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:46.017196 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:46.017262 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:46.017427 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.017894 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.019371 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.019982 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.020055 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:46.020063 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:46.020267 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 14:47:46.020308 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:47:46.020604 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:46.016772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:43835 (local address 127.18.96.129:36077)
0928 14:47:46.017037 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:46.017047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:46.017134 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:46.017279 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:46.017279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:46.017282 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:46.017358 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:46.017360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.017980 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.017981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:46.019258 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:46.019260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.020099 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.020100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:46.020182 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:46.020186 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:47:46.020465 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 14:47:46.020509 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":4,"thread_start_us":55,"threads_started":1}
I20260928 14:47:46.020551 20843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72cf236cdc94f97a96b5fcd7acd61e8" candidate_uuid: "b427479f46c740af8bde76d58b249be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da14f948644a9f83f1154ed8f74fdb" is_pre_election: true
I20260928 14:47:46.020735 20843 raft_consensus.cc:2454] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b427479f46c740af8bde76d58b249be4 in term 0.
I20260928 14:47:46.020934 20636 leader_election.cc:302] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da14f948644a9f83f1154ed8f74fdb, b427479f46c740af8bde76d58b249be4; no voters:
I20260928 14:47:46.021003 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72cf236cdc94f97a96b5fcd7acd61e8" candidate_uuid: "b427479f46c740af8bde76d58b249be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4" is_pre_election: true
I20260928 14:47:46.021090 21059 raft_consensus.cc:2781] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:46.021116 20983 raft_consensus.cc:2454] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b427479f46c740af8bde76d58b249be4 in term 0.
I20260928 14:47:46.021129 21059 raft_consensus.cc:482] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:46.021152 21059 raft_consensus.cc:3037] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:46.022053 21059 raft_consensus.cc:504] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:46.022197 21059 leader_election.cc:288] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [CANDIDATE]: Term 1 election: Requested vote from peers 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130:37117), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:46.022353 20843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72cf236cdc94f97a96b5fcd7acd61e8" candidate_uuid: "b427479f46c740af8bde76d58b249be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77da14f948644a9f83f1154ed8f74fdb"
I20260928 14:47:46.022387 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f72cf236cdc94f97a96b5fcd7acd61e8" candidate_uuid: "b427479f46c740af8bde76d58b249be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4"
I20260928 14:47:46.022459 20983 raft_consensus.cc:3037] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:46.022514 20843 raft_consensus.cc:3037] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:46.023303 20843 raft_consensus.cc:2454] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b427479f46c740af8bde76d58b249be4 in term 1.
I20260928 14:47:46.023599 20636 leader_election.cc:302] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da14f948644a9f83f1154ed8f74fdb, b427479f46c740af8bde76d58b249be4; no voters:
I20260928 14:47:46.023684 21059 raft_consensus.cc:2781] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:46.023814 21059 raft_consensus.cc:686] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 1 LEADER]: Becoming Leader. State: Replica: b427479f46c740af8bde76d58b249be4, State: Running, Role: LEADER
I20260928 14:47:46.023898 21059 consensus_queue.cc:237] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:46.024391 20983 raft_consensus.cc:2454] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b427479f46c740af8bde76d58b249be4 in term 1.
I20260928 14:47:46.024659 20546 catalog_manager.cc:5495] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b427479f46c740af8bde76d58b249be4 (127.18.96.129). New cstate: current_term: 1 leader_uuid: "b427479f46c740af8bde76d58b249be4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } health_report { overall_health: UNKNOWN } } }
I20260928 14:47:46.033347 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:46.030805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.18.96.129:36905)
0928 14:47:46.030848 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:46.030850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:46.030859 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:47:46.030916 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:46.030917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:46.030936 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:46.031020 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:46.031126 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.031705 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.032523 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.032731 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.032810 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:46.032981 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:46.033104 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:46.033187 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:46.033237 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 14:47:46.033258 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:47:46.038720 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:46.035935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:37117 (local address 127.18.96.129:37779)
0928 14:47:46.036085 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:46.036096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:46.036177 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:46.036304 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:46.036305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:46.036306 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:47:46.036369 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:46.036371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.037088 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.037089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:46.037759 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:46.037761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.038186 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.038187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:46.038276 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:46.038290 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:47:46.038512 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:47:46.038515 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:47:46.038568 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:47:46.038615 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260928 14:47:46.038692 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:46.035988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:37779 (local address 127.18.96.130:37117)
0928 14:47:46.036036 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:46.036039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:46.036162 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:47:46.036237 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:46.036237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:46.036247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:46.036292 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:46.036450 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.037013 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.037841 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.038072 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.038158 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:46.038406 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:46.038444 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:46.038510 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:46.038573 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 14:47:46.038593 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:47:46.041253 20843 raft_consensus.cc:1261] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Refusing update from remote peer b427479f46c740af8bde76d58b249be4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:46.041532 20983 raft_consensus.cc:1261] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Refusing update from remote peer b427479f46c740af8bde76d58b249be4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:46.041612 21100 consensus_queue.cc:1035] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:46.041730 21059 consensus_queue.cc:1035] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:46.050002 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:46.047569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:51437 (local address 127.18.96.129:36905)
0928 14:47:46.047631 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:46.047633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:46.047640 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:47:46.047662 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:46.047663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:46.047676 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:46.047731 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:46.047847 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.048478 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.049224 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.049526 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.049595 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:46.049769 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:46.049806 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:46.049849 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:46.049927 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:47:46.049948 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:47:46.050041 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:46.047405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36905 (local address 127.18.96.130:51437)
0928 14:47:46.047444 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:46.047457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:46.047509 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:46.047732 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:46.047733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:46.047734 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:47:46.047783 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:46.047784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.048568 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.048569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:46.049119 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:46.049121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:46.049598 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:46.049600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:46.049669 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:46.049674 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:47:46.049869 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:47:46.049870 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:47:46.049916 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 14:47:46.049960 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 14:47:46.050622 21078 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:47:47.576042 20545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:47.576166 20545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:47.577096 20545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab: Sending DeleteTablet for 3 replicas of tablet f72cf236cdc94f97a96b5fcd7acd61e8
I20260928 14:47:47.577389 20683 tablet_service.cc:1514] Processing DeleteTablet for tablet f72cf236cdc94f97a96b5fcd7acd61e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:47 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:60344
I20260928 14:47:47.577400 20823 tablet_service.cc:1514] Processing DeleteTablet for tablet f72cf236cdc94f97a96b5fcd7acd61e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:47 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:48888
I20260928 14:47:47.577430 20963 tablet_service.cc:1514] Processing DeleteTablet for tablet f72cf236cdc94f97a96b5fcd7acd61e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:47 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:41956
I20260928 14:47:47.577615 21130 tablet_replica.cc:330] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: stopping tablet replica
I20260928 14:47:47.577677 21130 raft_consensus.cc:2229] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:47:47.577778 21130 raft_consensus.cc:2258] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:47.577890 21131 tablet_replica.cc:330] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: stopping tablet replica
I20260928 14:47:47.577950 21131 raft_consensus.cc:2229] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:47.578004 21131 raft_consensus.cc:2258] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:47.578148 21129 tablet_replica.cc:330] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: stopping tablet replica
I20260928 14:47:47.578207 21129 raft_consensus.cc:2229] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:47.578248 21129 raft_consensus.cc:2258] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:47.578433 21131 ts_tablet_manager.cc:1905] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:47.579656 21130 ts_tablet_manager.cc:1905] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:47.579945 21131 ts_tablet_manager.cc:1918] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:47:47.580066 21131 log.cc:1196] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/f72cf236cdc94f97a96b5fcd7acd61e8
I20260928 14:47:47.580368 21131 ts_tablet_manager.cc:1939] T f72cf236cdc94f97a96b5fcd7acd61e8 P 3afaba5fae404ff6a181dda9104b7ba4: Deleting consensus metadata
I20260928 14:47:47.580746 20530 catalog_manager.cc:4841] TS 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835): tablet f72cf236cdc94f97a96b5fcd7acd61e8 (table test-table [id=ced25a20bb344771a27e432e4d3af9f6]) successfully deleted
I20260928 14:47:47.581305 21130 ts_tablet_manager.cc:1918] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:47:47.581357 21130 log.cc:1196] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/f72cf236cdc94f97a96b5fcd7acd61e8
I20260928 14:47:47.581527 21130 ts_tablet_manager.cc:1939] T f72cf236cdc94f97a96b5fcd7acd61e8 P b427479f46c740af8bde76d58b249be4: Deleting consensus metadata
I20260928 14:47:47.581777 21129 ts_tablet_manager.cc:1905] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:47.582031 20532 catalog_manager.cc:4841] TS b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905): tablet f72cf236cdc94f97a96b5fcd7acd61e8 (table test-table [id=ced25a20bb344771a27e432e4d3af9f6]) successfully deleted
I20260928 14:47:47.583267 21129 ts_tablet_manager.cc:1918] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:47:47.583344 21129 log.cc:1196] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/f72cf236cdc94f97a96b5fcd7acd61e8
I20260928 14:47:47.583575 21129 ts_tablet_manager.cc:1939] T f72cf236cdc94f97a96b5fcd7acd61e8 P 77da14f948644a9f83f1154ed8f74fdb: Deleting consensus metadata
I20260928 14:47:47.584002 20532 catalog_manager.cc:4841] TS 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130:37117): tablet f72cf236cdc94f97a96b5fcd7acd61e8 (table test-table [id=ced25a20bb344771a27e432e4d3af9f6]) successfully deleted
Sep 28 14:47:47 dist-test-slave-j968 krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:47:47 dist-test-slave-j968 krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:47.601979 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:47.594955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.18.96.190:35287)
0928 14:47:47.595199 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:47.595201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:47.595210 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:47:47.595242 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:47.595243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:47.595253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:47.595320 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:47.595843 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:47.596429 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:47.597147 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:47.597335 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:47.597383 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:47.599478 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:47.599480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:47.599480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:47.599487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:47.601037 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:47.601335 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:47.601336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:47.601336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:47.601385 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:47.601528 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:47.601529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:47.601529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:47.601620 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:47.601689 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:47.601812 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 14:47:47.601889 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":190,"threads_started":1}
I20260928 14:47:47.603714 20545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:47:47.603900 20545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:47.605327 20823 tablet_service.cc:1467] Processing CreateTablet for tablet 54cbac34cdb34bb09ac7cb0f220c653f (DEFAULT_TABLE table=test-table [id=d226a66d981b4150ad2679d821c37c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:47.605324 20683 tablet_service.cc:1467] Processing CreateTablet for tablet 54cbac34cdb34bb09ac7cb0f220c653f (DEFAULT_TABLE table=test-table [id=d226a66d981b4150ad2679d821c37c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:47.605365 20963 tablet_service.cc:1467] Processing CreateTablet for tablet 54cbac34cdb34bb09ac7cb0f220c653f (DEFAULT_TABLE table=test-table [id=d226a66d981b4150ad2679d821c37c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:47.605465 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54cbac34cdb34bb09ac7cb0f220c653f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:47.605489 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54cbac34cdb34bb09ac7cb0f220c653f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:47.605515 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54cbac34cdb34bb09ac7cb0f220c653f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:47.606832 21142 tablet_bootstrap.cc:492] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: Bootstrap starting.
I20260928 14:47:47.606997 21140 tablet_bootstrap.cc:492] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: Bootstrap starting.
I20260928 14:47:47.607221 21142 tablet_bootstrap.cc:654] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:47.607401 21140 tablet_bootstrap.cc:654] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:47.607542 21141 tablet_bootstrap.cc:492] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: Bootstrap starting.
I20260928 14:47:47.607815 21142 tablet_bootstrap.cc:492] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: No bootstrap required, opened a new log
I20260928 14:47:47.607897 21142 ts_tablet_manager.cc:1397] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:47.607955 21141 tablet_bootstrap.cc:654] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:47.608019 21142 raft_consensus.cc:348] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.608134 21142 raft_consensus.cc:374] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:47.608150 21140 tablet_bootstrap.cc:492] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: No bootstrap required, opened a new log
I20260928 14:47:47.608178 21142 raft_consensus.cc:729] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b427479f46c740af8bde76d58b249be4, State: Initialized, Role: FOLLOWER
I20260928 14:47:47.608219 21140 ts_tablet_manager.cc:1397] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:47.608256 21142 consensus_queue.cc:260] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.608351 21140 raft_consensus.cc:348] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.608465 21140 raft_consensus.cc:374] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:47.608536 21140 raft_consensus.cc:729] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afaba5fae404ff6a181dda9104b7ba4, State: Initialized, Role: FOLLOWER
I20260928 14:47:47.608635 21140 consensus_queue.cc:260] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.608810 21140 ts_tablet_manager.cc:1428] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:47:47.608382 21142 ts_tablet_manager.cc:1428] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:47.609074 21141 tablet_bootstrap.cc:492] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: No bootstrap required, opened a new log
I20260928 14:47:47.609131 21141 ts_tablet_manager.cc:1397] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:47:47.609233 21141 raft_consensus.cc:348] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.609393 21141 raft_consensus.cc:374] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:47.609416 21141 raft_consensus.cc:729] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da14f948644a9f83f1154ed8f74fdb, State: Initialized, Role: FOLLOWER
I20260928 14:47:47.609474 21141 consensus_queue.cc:260] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.609637 21141 ts_tablet_manager.cc:1428] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:47.718978 21116 raft_consensus.cc:482] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:47.719079 21116 raft_consensus.cc:504] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.719264 21116 leader_election.cc:288] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:47.719501 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54cbac34cdb34bb09ac7cb0f220c653f" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b427479f46c740af8bde76d58b249be4" is_pre_election: true
I20260928 14:47:47.719547 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54cbac34cdb34bb09ac7cb0f220c653f" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4" is_pre_election: true
I20260928 14:47:47.719638 20703 raft_consensus.cc:2454] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 0.
I20260928 14:47:47.719631 20983 raft_consensus.cc:2454] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 0.
I20260928 14:47:47.719826 20776 leader_election.cc:302] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da14f948644a9f83f1154ed8f74fdb, b427479f46c740af8bde76d58b249be4; no voters:
I20260928 14:47:47.719985 21116 raft_consensus.cc:2781] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:47.720036 21116 raft_consensus.cc:482] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:47.720049 21116 raft_consensus.cc:3037] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:47.720939 21116 raft_consensus.cc:504] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.721092 21116 leader_election.cc:288] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 election: Requested vote from peers b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:47.721223 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54cbac34cdb34bb09ac7cb0f220c653f" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b427479f46c740af8bde76d58b249be4"
I20260928 14:47:47.721297 20703 raft_consensus.cc:3037] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:47.721289 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54cbac34cdb34bb09ac7cb0f220c653f" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4"
I20260928 14:47:47.721372 20983 raft_consensus.cc:3037] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:47.721942 20983 raft_consensus.cc:2454] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 1.
I20260928 14:47:47.722008 20703 raft_consensus.cc:2454] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 1.
I20260928 14:47:47.722126 20774 leader_election.cc:302] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3afaba5fae404ff6a181dda9104b7ba4, 77da14f948644a9f83f1154ed8f74fdb; no voters:
I20260928 14:47:47.722213 21116 raft_consensus.cc:2781] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:47.722263 21116 raft_consensus.cc:686] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 1 LEADER]: Becoming Leader. State: Replica: 77da14f948644a9f83f1154ed8f74fdb, State: Running, Role: LEADER
I20260928 14:47:47.722322 21116 consensus_queue.cc:237] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.722952 20546 catalog_manager.cc:5495] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130). New cstate: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } health_report { overall_health: UNKNOWN } } }
I20260928 14:47:47.755993 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:47.753521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.18.96.130:37117)
0928 14:47:47.753693 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:47.753694 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:47.753707 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:47:47.753725 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:47.753725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:47.753735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:47.753809 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:47.753969 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:47.754449 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:47.755233 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:47.755444 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:47.755517 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:47.755674 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:47.755715 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:47.755774 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:47.755844 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 14:47:47.755896 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260928 14:47:47.757917 20983 raft_consensus.cc:1261] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Refusing update from remote peer 77da14f948644a9f83f1154ed8f74fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:47.757942 20703 raft_consensus.cc:1261] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Refusing update from remote peer 77da14f948644a9f83f1154ed8f74fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:47.758148 21111 consensus_queue.cc:1035] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:47.758246 21062 consensus_queue.cc:1035] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:47.768613 20545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:47.768729 20545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:47.769377 20545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab: Sending DeleteTablet for 3 replicas of tablet 54cbac34cdb34bb09ac7cb0f220c653f
I20260928 14:47:47.769618 20963 tablet_service.cc:1514] Processing DeleteTablet for tablet 54cbac34cdb34bb09ac7cb0f220c653f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:47 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:41956
I20260928 14:47:47.769627 20683 tablet_service.cc:1514] Processing DeleteTablet for tablet 54cbac34cdb34bb09ac7cb0f220c653f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:47 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:60344
I20260928 14:47:47.769701 20823 tablet_service.cc:1514] Processing DeleteTablet for tablet 54cbac34cdb34bb09ac7cb0f220c653f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:47 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:48888
I20260928 14:47:47.769791 21131 tablet_replica.cc:330] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: stopping tablet replica
I20260928 14:47:47.769793 21129 tablet_replica.cc:330] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: stopping tablet replica
I20260928 14:47:47.769852 21131 raft_consensus.cc:2229] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:47.769858 21129 raft_consensus.cc:2229] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:47:47.769902 21131 raft_consensus.cc:2258] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:47.769923 21129 raft_consensus.cc:2258] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:47.769979 21130 tablet_replica.cc:330] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: stopping tablet replica
I20260928 14:47:47.770038 21130 raft_consensus.cc:2229] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:47.769950 20545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:47:47.770097 21130 raft_consensus.cc:2258] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:47:47.770169 20545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:47.770399 21130 ts_tablet_manager.cc:1905] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:47.770568 21131 ts_tablet_manager.cc:1905] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:47.771454 20823 tablet_service.cc:1467] Processing CreateTablet for tablet fb13bc7278aa4492bc0c22101267cf4f (DEFAULT_TABLE table=test-table [id=e41a92c28a564ee8a4c4cd0a1d60d5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:47.771453 20683 tablet_service.cc:1467] Processing CreateTablet for tablet fb13bc7278aa4492bc0c22101267cf4f (DEFAULT_TABLE table=test-table [id=e41a92c28a564ee8a4c4cd0a1d60d5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:47.771570 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb13bc7278aa4492bc0c22101267cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:47.771570 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb13bc7278aa4492bc0c22101267cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:47.772233 20963 tablet_service.cc:1467] Processing CreateTablet for tablet fb13bc7278aa4492bc0c22101267cf4f (DEFAULT_TABLE table=test-table [id=e41a92c28a564ee8a4c4cd0a1d60d5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:47.772435 21129 ts_tablet_manager.cc:1905] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:47.772881 21141 tablet_bootstrap.cc:492] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: Bootstrap starting.
I20260928 14:47:47.773201 21141 tablet_bootstrap.cc:654] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:47.773732 21141 tablet_bootstrap.cc:492] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: No bootstrap required, opened a new log
I20260928 14:47:47.773777 21141 ts_tablet_manager.cc:1397] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:47.773849 21129 ts_tablet_manager.cc:1918] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:47:47.773854 21131 ts_tablet_manager.cc:1918] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:47:47.773900 21129 log.cc:1196] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/54cbac34cdb34bb09ac7cb0f220c653f
I20260928 14:47:47.773906 21131 log.cc:1196] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/54cbac34cdb34bb09ac7cb0f220c653f
I20260928 14:47:47.773911 21141 raft_consensus.cc:348] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.773985 21141 raft_consensus.cc:374] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:47.774010 21141 raft_consensus.cc:729] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77da14f948644a9f83f1154ed8f74fdb, State: Initialized, Role: FOLLOWER
I20260928 14:47:47.774057 21141 consensus_queue.cc:260] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.774147 21131 ts_tablet_manager.cc:1939] T 54cbac34cdb34bb09ac7cb0f220c653f P 3afaba5fae404ff6a181dda9104b7ba4: Deleting consensus metadata
I20260928 14:47:47.774178 21129 ts_tablet_manager.cc:1939] T 54cbac34cdb34bb09ac7cb0f220c653f P 77da14f948644a9f83f1154ed8f74fdb: Deleting consensus metadata
I20260928 14:47:47.774295 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb13bc7278aa4492bc0c22101267cf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:47.774165 21141 ts_tablet_manager.cc:1428] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:47.774525 20532 catalog_manager.cc:4841] TS 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130:37117): tablet 54cbac34cdb34bb09ac7cb0f220c653f (table test-table [id=d226a66d981b4150ad2679d821c37c35]) successfully deleted
I20260928 14:47:47.775230 21140 tablet_bootstrap.cc:492] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: Bootstrap starting.
I20260928 14:47:47.775548 21130 ts_tablet_manager.cc:1918] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:47:47.775609 21130 log.cc:1196] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/54cbac34cdb34bb09ac7cb0f220c653f
I20260928 14:47:47.775605 21140 tablet_bootstrap.cc:654] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:47.775789 21130 ts_tablet_manager.cc:1939] T 54cbac34cdb34bb09ac7cb0f220c653f P b427479f46c740af8bde76d58b249be4: Deleting consensus metadata
I20260928 14:47:47.776136 20532 catalog_manager.cc:4841] TS b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905): tablet 54cbac34cdb34bb09ac7cb0f220c653f (table test-table [id=d226a66d981b4150ad2679d821c37c35]) successfully deleted
I20260928 14:47:47.776217 21140 tablet_bootstrap.cc:492] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: No bootstrap required, opened a new log
I20260928 14:47:47.776266 21140 ts_tablet_manager.cc:1397] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:47.776413 21140 raft_consensus.cc:348] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.776494 21140 raft_consensus.cc:374] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:47.776522 21140 raft_consensus.cc:729] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afaba5fae404ff6a181dda9104b7ba4, State: Initialized, Role: FOLLOWER
I20260928 14:47:47.776589 21140 consensus_queue.cc:260] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.776616 21142 tablet_bootstrap.cc:492] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4: Bootstrap starting.
I20260928 14:47:47.776693 21140 ts_tablet_manager.cc:1428] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:47.776840 20530 catalog_manager.cc:4841] TS 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835): tablet 54cbac34cdb34bb09ac7cb0f220c653f (table test-table [id=d226a66d981b4150ad2679d821c37c35]) successfully deleted
I20260928 14:47:47.776925 21142 tablet_bootstrap.cc:654] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:47.777532 21142 tablet_bootstrap.cc:492] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4: No bootstrap required, opened a new log
I20260928 14:47:47.777578 21142 ts_tablet_manager.cc:1397] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:47.777748 21142 raft_consensus.cc:348] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.777804 21142 raft_consensus.cc:374] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:47.777848 21142 raft_consensus.cc:729] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b427479f46c740af8bde76d58b249be4, State: Initialized, Role: FOLLOWER
I20260928 14:47:47.777894 21142 consensus_queue.cc:260] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:47.777987 21142 ts_tablet_manager.cc:1428] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:48.057972 21116 raft_consensus.cc:482] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:48.058068 21116 raft_consensus.cc:504] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:48.058225 21116 leader_election.cc:288] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:48.058422 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb13bc7278aa4492bc0c22101267cf4f" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4" is_pre_election: true
I20260928 14:47:48.058506 20983 raft_consensus.cc:2454] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 0.
I20260928 14:47:48.058429 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb13bc7278aa4492bc0c22101267cf4f" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b427479f46c740af8bde76d58b249be4" is_pre_election: true
I20260928 14:47:48.058576 20703 raft_consensus.cc:2454] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 0.
I20260928 14:47:48.058697 20774 leader_election.cc:302] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3afaba5fae404ff6a181dda9104b7ba4, 77da14f948644a9f83f1154ed8f74fdb; no voters:
I20260928 14:47:48.058815 21116 raft_consensus.cc:2781] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:48.058884 21116 raft_consensus.cc:482] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:48.058908 21116 raft_consensus.cc:3037] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:48.059618 21116 raft_consensus.cc:504] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:48.059769 21116 leader_election.cc:288] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 election: Requested vote from peers b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905), 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835)
I20260928 14:47:48.059933 20703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb13bc7278aa4492bc0c22101267cf4f" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b427479f46c740af8bde76d58b249be4"
I20260928 14:47:48.060020 20703 raft_consensus.cc:3037] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:48.060017 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb13bc7278aa4492bc0c22101267cf4f" candidate_uuid: "77da14f948644a9f83f1154ed8f74fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3afaba5fae404ff6a181dda9104b7ba4"
I20260928 14:47:48.060139 20983 raft_consensus.cc:3037] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:48.060644 20703 raft_consensus.cc:2454] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 1.
I20260928 14:47:48.060776 20983 raft_consensus.cc:2454] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77da14f948644a9f83f1154ed8f74fdb in term 1.
I20260928 14:47:48.060887 20776 leader_election.cc:302] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77da14f948644a9f83f1154ed8f74fdb, b427479f46c740af8bde76d58b249be4; no voters:
I20260928 14:47:48.060976 21116 raft_consensus.cc:2781] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:48.061038 21116 raft_consensus.cc:686] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 1 LEADER]: Becoming Leader. State: Replica: 77da14f948644a9f83f1154ed8f74fdb, State: Running, Role: LEADER
I20260928 14:47:48.061090 21116 consensus_queue.cc:237] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } }
I20260928 14:47:48.061738 20545 catalog_manager.cc:5495] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130). New cstate: current_term: 1 leader_uuid: "77da14f948644a9f83f1154ed8f74fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77da14f948644a9f83f1154ed8f74fdb" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 37117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 } health_report { overall_health: UNKNOWN } } }
I20260928 14:47:48.130060 20983 raft_consensus.cc:1261] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Refusing update from remote peer 77da14f948644a9f83f1154ed8f74fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:48.130060 20703 raft_consensus.cc:1261] T fb13bc7278aa4492bc0c22101267cf4f P b427479f46c740af8bde76d58b249be4 [term 1 FOLLOWER]: Refusing update from remote peer 77da14f948644a9f83f1154ed8f74fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:48.130321 21062 consensus_queue.cc:1035] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3afaba5fae404ff6a181dda9104b7ba4" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:48.130416 21153 consensus_queue.cc:1035] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b427479f46c740af8bde76d58b249be4" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:47:48.140249 21109 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:47:49.657838 20545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:49.657938 20545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:49.658717 20545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29039138a55b4c35904b7f402a765aab: Sending DeleteTablet for 3 replicas of tablet fb13bc7278aa4492bc0c22101267cf4f
I20260928 14:47:49.658946 20683 tablet_service.cc:1514] Processing DeleteTablet for tablet fb13bc7278aa4492bc0c22101267cf4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:49 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:60344
I20260928 14:47:49.658986 20963 tablet_service.cc:1514] Processing DeleteTablet for tablet fb13bc7278aa4492bc0c22101267cf4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:49 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:41956
I20260928 14:47:49.659001 20823 tablet_service.cc:1514] Processing DeleteTablet for tablet fb13bc7278aa4492bc0c22101267cf4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:49 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:48888
I20260928 14:47:49.659106 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 20618
I20260928 14:47:49.660887 21181 tablet_replica.cc:330] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: stopping tablet replica
I20260928 14:47:49.660945 21181 raft_consensus.cc:2229] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:47:49.661010 21181 raft_consensus.cc:2258] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:49.661124 21179 tablet_replica.cc:330] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: stopping tablet replica
I20260928 14:47:49.661183 21179 raft_consensus.cc:2229] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:49.661221 21179 raft_consensus.cc:2258] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:49.661240 21181 ts_tablet_manager.cc:1905] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:49.661398 21179 ts_tablet_manager.cc:1905] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:49.663137 21179 ts_tablet_manager.cc:1918] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:47:49.663137 21181 ts_tablet_manager.cc:1918] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:47:49.663199 21179 log.cc:1196] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/fb13bc7278aa4492bc0c22101267cf4f
I20260928 14:47:49.663199 21181 log.cc:1196] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/fb13bc7278aa4492bc0c22101267cf4f
I20260928 14:47:49.663424 21179 ts_tablet_manager.cc:1939] T fb13bc7278aa4492bc0c22101267cf4f P 3afaba5fae404ff6a181dda9104b7ba4: Deleting consensus metadata
I20260928 14:47:49.663462 21181 ts_tablet_manager.cc:1939] T fb13bc7278aa4492bc0c22101267cf4f P 77da14f948644a9f83f1154ed8f74fdb: Deleting consensus metadata
I20260928 14:47:49.663763 20530 catalog_manager.cc:4841] TS 3afaba5fae404ff6a181dda9104b7ba4 (127.18.96.131:43835): tablet fb13bc7278aa4492bc0c22101267cf4f (table test-table [id=e41a92c28a564ee8a4c4cd0a1d60d5bc]) successfully deleted
I20260928 14:47:49.663769 20532 catalog_manager.cc:4841] TS 77da14f948644a9f83f1154ed8f74fdb (127.18.96.130:37117): tablet fb13bc7278aa4492bc0c22101267cf4f (table test-table [id=e41a92c28a564ee8a4c4cd0a1d60d5bc]) successfully deleted
W20260928 14:47:49.665522 20532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:36905 (error 108)
I20260928 14:47:49.665771 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 20758
I20260928 14:47:49.666824 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:49.665909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36905 (local address 127.0.0.1:48082)
0928 14:47:49.666750 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:49.666792 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:36905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":63,"threads_started":1}
W20260928 14:47:49.666939 20532 catalog_manager.cc:4584] TS b427479f46c740af8bde76d58b249be4 (127.18.96.129:36905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb13bc7278aa4492bc0c22101267cf4f: Network error: Client connection negotiation failed: client connection to 127.18.96.129:36905: connect: Connection refused (error 111)
I20260928 14:47:49.672349 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 20898
I20260928 14:47:49.677917 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 20514
2026-09-28T14:47:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5797 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:47:49 dist-test-slave-j968 krb5kdc[21186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:47:49 dist-test-slave-j968 krb5kdc[21186](info): set up 2 sockets
Sep 28 14:47:49 dist-test-slave-j968 krb5kdc[21186](info): commencing operation
krb5kdc: starting...
W20260928 14:47:50.724506 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:47:50 dist-test-slave-j968 krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:47:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:47:50.843534 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35489
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:35489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:50.925048 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:50.925271 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:50.925294 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:50.926438 21199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:47:50.926483 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:50.926498 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:50.926512 21199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:47:50.926523 21199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:47:50.927969 21199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:35489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:50.928172 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:50.928350 21199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:50.933336 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:50.933362 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:50.933410 21199 server_base.cc:1034] running on GCE node
W20260928 14:47:50.933365 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:50.933907 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:50.934243 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:50.935446 21199 hybrid_clock.cc:648] HybridClock initialized: now 1790606870935428 us; error 27 us; skew 500 ppm
Sep 28 14:47:50 dist-test-slave-j968 krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606870, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:50.938087 21199 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:47:50.938525 21199 webserver.cc:460] Webserver started at http://127.18.96.190:33389/ using document root <none> and password file <none>
I20260928 14:47:50.938781 21199 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:50.938848 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:50.938997 21199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:50.940155 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "5438f80fe9544abd9475d267733adbdc"
format_stamp: "Formatted at 2026-09-28 14:47:50 on dist-test-slave-j968"
server_key: "dc8b3a0a80f5d6f29d1749052d114aa3"
server_key_iv: "3f0b51dc1bbde68ed516fda5e2073cd3"
server_key_version: "encryptionkey@0"
I20260928 14:47:50.940533 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "5438f80fe9544abd9475d267733adbdc"
format_stamp: "Formatted at 2026-09-28 14:47:50 on dist-test-slave-j968"
server_key: "dc8b3a0a80f5d6f29d1749052d114aa3"
server_key_iv: "3f0b51dc1bbde68ed516fda5e2073cd3"
server_key_version: "encryptionkey@0"
I20260928 14:47:50.941900 21199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:47:50.942622 21214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:50.942794 21199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:50.942899 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "5438f80fe9544abd9475d267733adbdc"
format_stamp: "Formatted at 2026-09-28 14:47:50 on dist-test-slave-j968"
server_key: "dc8b3a0a80f5d6f29d1749052d114aa3"
server_key_iv: "3f0b51dc1bbde68ed516fda5e2073cd3"
server_key_version: "encryptionkey@0"
I20260928 14:47:50.942973 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:50.966861 21199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:50.969393 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:50.969597 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:50.973659 21199 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:35489
I20260928 14:47:50.973695 21266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:35489 every 8 connection(s)
I20260928 14:47:50.974107 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:47:50.974983 21267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:50.977258 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc: Bootstrap starting.
I20260928 14:47:50.977807 21267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:50.978035 21267 log.cc:824] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:50.978638 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 21199
I20260928 14:47:50.978730 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:47:50.978852 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc: No bootstrap required, opened a new log
I20260928 14:47:50.978899 18818 external_mini_cluster.cc:1428] Setting key f6a11020aadffcd8b73d632f073b6089
I20260928 14:47:50.981364 21267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5438f80fe9544abd9475d267733adbdc" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35489 } }
I20260928 14:47:50.981555 21267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:50.981602 21267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5438f80fe9544abd9475d267733adbdc, State: Initialized, Role: FOLLOWER
I20260928 14:47:50.981729 21267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5438f80fe9544abd9475d267733adbdc" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35489 } }
I20260928 14:47:50.981827 21267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:47:50.981884 21267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:47:50.981932 21267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:50.982477 21267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5438f80fe9544abd9475d267733adbdc" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35489 } }
I20260928 14:47:50.982599 21267 leader_election.cc:302] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5438f80fe9544abd9475d267733adbdc; no voters:
I20260928 14:47:50.982760 21267 leader_election.cc:288] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:47:50.982788 21272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:50.982923 21272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [term 1 LEADER]: Becoming Leader. State: Replica: 5438f80fe9544abd9475d267733adbdc, State: Running, Role: LEADER
I20260928 14:47:50.983052 21267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:47:50.983070 21272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5438f80fe9544abd9475d267733adbdc" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35489 } }
I20260928 14:47:50.983395 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5438f80fe9544abd9475d267733adbdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5438f80fe9544abd9475d267733adbdc" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35489 } } }
I20260928 14:47:50.983379 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5438f80fe9544abd9475d267733adbdc. Latest consensus state: current_term: 1 leader_uuid: "5438f80fe9544abd9475d267733adbdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5438f80fe9544abd9475d267733adbdc" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35489 } } }
I20260928 14:47:50.983623 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:50.983645 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:50.984578 21281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:47:50.985251 21281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:47:50.987087 21281 catalog_manager.cc:1347] Generated new cluster ID: 07412f5328f44c1da8bb5479767c396d
I20260928 14:47:50.987149 21281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:47:50 dist-test-slave-j968 krb5kdc[21186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:50.991266 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:50.979555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.18.96.190:35489)
0928 14:47:50.979763 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:50.979771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:50.979798 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:47:50.980322 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:50.980330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:50.980380 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:50.980534 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:50.981421 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:50.983048 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:50.984327 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:50.985021 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:50.985167 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:50.987980 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:50.987984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:50.987985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:50.987996 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:50.989734 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:50.990140 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:50.990143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:50.990143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:50.990233 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:50.990448 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:50.990449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:50.990449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:50.990695 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:50.990874 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:50.990948 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 14:47:50.991095 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260928 14:47:51.009157 21281 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:47:51.009812 21281 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:47:51.015118 21281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5438f80fe9544abd9475d267733adbdc: Generated new TSK 0
I20260928 14:47:51.015296 21281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:47:51.047883 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35489
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:51.142616 21293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:51.142856 21293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:51.142894 21293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:51.144236 21293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:51.144296 21293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:51.144341 21293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:47:51.145881 21293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:51.146147 21293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:51.146406 21293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:51.153043 21301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:51.153226 21298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:51.153482 21293 server_base.cc:1034] running on GCE node
W20260928 14:47:51.155014 21299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:51.155364 21293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:51.155697 21293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:51.156857 21293 hybrid_clock.cc:648] HybridClock initialized: now 1790606871156845 us; error 32 us; skew 500 ppm
Sep 28 14:47:51 dist-test-slave-j968 krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606871, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:51.160181 21293 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:47:51.160815 21293 webserver.cc:460] Webserver started at http://127.18.96.129:34225/ using document root <none> and password file <none>
I20260928 14:47:51.161103 21293 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:51.161216 21293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:51.161374 21293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:51.162973 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "e3cfef3bca9640659ebc0582c519cc10"
format_stamp: "Formatted at 2026-09-28 14:47:51 on dist-test-slave-j968"
server_key: "4d1919e8491e347b7ac843189b8b3df9"
server_key_iv: "10dbf14f4e68b7dd2357a1fc35315427"
server_key_version: "encryptionkey@0"
I20260928 14:47:51.163443 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "e3cfef3bca9640659ebc0582c519cc10"
format_stamp: "Formatted at 2026-09-28 14:47:51 on dist-test-slave-j968"
server_key: "4d1919e8491e347b7ac843189b8b3df9"
server_key_iv: "10dbf14f4e68b7dd2357a1fc35315427"
server_key_version: "encryptionkey@0"
I20260928 14:47:51.165359 21293 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 14:47:51.166347 21308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.166499 21293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:47:51.166580 21293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "e3cfef3bca9640659ebc0582c519cc10"
format_stamp: "Formatted at 2026-09-28 14:47:51 on dist-test-slave-j968"
server_key: "4d1919e8491e347b7ac843189b8b3df9"
server_key_iv: "10dbf14f4e68b7dd2357a1fc35315427"
server_key_version: "encryptionkey@0"
I20260928 14:47:51.166671 21293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:51.206183 21293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:51.208240 21293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:51.208415 21293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:51.208667 21293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:51.209002 21293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:51.209056 21293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.209091 21293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:51.209139 21293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.214864 21293 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:45715
I20260928 14:47:51.214947 21421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:45715 every 8 connection(s)
I20260928 14:47:51.215282 21293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:47:51 dist-test-slave-j968 krb5kdc[21186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606871, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:51.224177 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 21293
I20260928 14:47:51.224296 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:47:51.224527 18818 external_mini_cluster.cc:1428] Setting key 673333c263341e5150e26932b1a117d3
I20260928 14:47:51.225857 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:51.216122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:51045 (local address 127.18.96.190:35489)
0928 14:47:51.216168 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:51.216171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:51.216359 (+ 188us) server_negotiation.cc:407] Connection header received
0928 14:47:51.216910 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:51.216912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:51.216926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:51.216988 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:51.217995 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.218521 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.219581 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.219791 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.219914 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:51.222550 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:51.222551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:51.222551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:51.222558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:51.224090 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:51.224689 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:51.224690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:51.224691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:51.224774 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:51.225097 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:51.225099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:51.225099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:51.225228 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:51.225301 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:51.225701 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 14:47:51.225790 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:47:51.226156 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:51.216083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35489 (local address 127.18.96.129:51045)
0928 14:47:51.216223 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:51.216273 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:51.216801 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:51.217057 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:51.217061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:51.217657 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:51.217866 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:51.217871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.218622 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.218623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:51.219463 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:51.219465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.220045 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.220046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:51.220229 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:51.220890 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:51.220899 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:51.222458 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:51.224180 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:51.224189 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:51.224192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:51.224600 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:51.224853 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:51.224854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:51.224855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:51.225014 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:51.225323 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:51.225330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:51.225515 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 14:47:51.225669 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 14:47:51.225860 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260928 14:47:51.226724 21422 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35489
I20260928 14:47:51.226851 21422 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:51.227072 21422 heartbeater.cc:511] Master 127.18.96.190:35489 requested a full tablet report, sending...
I20260928 14:47:51.227648 21231 ts_manager.cc:194] Registered new tserver with Master: e3cfef3bca9640659ebc0582c519cc10 (127.18.96.129:45715)
I20260928 14:47:51.228446 21231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:51045
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:47:51.250872 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35489
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:51.332947 21427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:51.333150 21427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:51.333174 21427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:51.334354 21427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:51.334396 21427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:51.334422 21427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:47:51.335961 21427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:51.336200 21427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:51.336460 21427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:51.341969 21433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:51.341997 21435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:51.341976 21432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:51.342224 21427 server_base.cc:1034] running on GCE node
I20260928 14:47:51.342466 21427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:51.342665 21427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:51.343834 21427 hybrid_clock.cc:648] HybridClock initialized: now 1790606871343826 us; error 39 us; skew 500 ppm
Sep 28 14:47:51 dist-test-slave-j968 krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606871, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:51.346467 21427 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:47:51.347007 21427 webserver.cc:460] Webserver started at http://127.18.96.130:40837/ using document root <none> and password file <none>
I20260928 14:47:51.347237 21427 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:51.347311 21427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:51.347474 21427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:51.348665 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "dd33e0a140d64044855050bcf98245af"
format_stamp: "Formatted at 2026-09-28 14:47:51 on dist-test-slave-j968"
server_key: "a42e1b9b324945f05a5256e07c826b29"
server_key_iv: "fb73f3673bed20009ba0683810735cbb"
server_key_version: "encryptionkey@0"
I20260928 14:47:51.349054 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "dd33e0a140d64044855050bcf98245af"
format_stamp: "Formatted at 2026-09-28 14:47:51 on dist-test-slave-j968"
server_key: "a42e1b9b324945f05a5256e07c826b29"
server_key_iv: "fb73f3673bed20009ba0683810735cbb"
server_key_version: "encryptionkey@0"
I20260928 14:47:51.350438 21427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:47:51.351323 21442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.351511 21427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.351620 21427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "dd33e0a140d64044855050bcf98245af"
format_stamp: "Formatted at 2026-09-28 14:47:51 on dist-test-slave-j968"
server_key: "a42e1b9b324945f05a5256e07c826b29"
server_key_iv: "fb73f3673bed20009ba0683810735cbb"
server_key_version: "encryptionkey@0"
I20260928 14:47:51.351714 21427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:51.374193 21427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:51.376235 21427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:51.376412 21427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:51.376665 21427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:51.376995 21427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:51.377059 21427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.377102 21427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:51.377127 21427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.382874 21427 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:33649
I20260928 14:47:51.382920 21555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:33649 every 8 connection(s)
I20260928 14:47:51.383327 21427 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:47:51.385922 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 21427
I20260928 14:47:51.386031 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:47:51.386217 18818 external_mini_cluster.cc:1428] Setting key 8e0431b118636fda70787cca56a84103
Sep 28 14:47:51 dist-test-slave-j968 krb5kdc[21186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606871, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:51.393952 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:51.384132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:43311 (local address 127.18.96.190:35489)
0928 14:47:51.384188 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:51.384191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:51.384371 (+ 180us) server_negotiation.cc:407] Connection header received
0928 14:47:51.384905 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:51.384906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:51.384915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:51.384979 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:51.385919 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.386624 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.388192 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.388455 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.388575 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:51.390772 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:51.390774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:51.390774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:51.390784 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:51.392570 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:51.392965 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:51.392966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:51.392967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:51.393037 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:51.393268 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:51.393269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:51.393270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:51.393428 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:51.393505 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:51.393760 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 14:47:51.393850 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:47:51.394037 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:51.384095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35489 (local address 127.18.96.130:43311)
0928 14:47:51.384254 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:51.384294 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:51.384820 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:51.385042 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:51.385046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:51.385611 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:51.385805 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:51.385810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.386698 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.386699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:51.388098 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:51.388101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.388565 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.388566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:51.388695 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:51.389264 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:51.389272 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:51.390665 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:51.392657 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:51.392663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:51.392665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:51.392886 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:51.393108 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:51.393109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:51.393109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:51.393203 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:51.393495 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:51.393499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:51.393629 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 14:47:51.393711 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:47:51.393837 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260928 14:47:51.394577 21556 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35489
I20260928 14:47:51.394691 21556 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:51.394922 21556 heartbeater.cc:511] Master 127.18.96.190:35489 requested a full tablet report, sending...
I20260928 14:47:51.395335 21231 ts_manager.cc:194] Registered new tserver with Master: dd33e0a140d64044855050bcf98245af (127.18.96.130:33649)
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
I20260928 14:47:51.395715 21231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:43311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:47:51.414475 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35489
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:51.496984 21561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:51.497213 21561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:51.497252 21561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:51.498411 21561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:51.498476 21561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:51.498519 21561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:47:51.500151 21561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:51.500393 21561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:51.500622 21561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:51.505587 21566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:51.505626 21567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:51.505584 21569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:51.505877 21561 server_base.cc:1034] running on GCE node
I20260928 14:47:51.506081 21561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:51.506276 21561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:51.507467 21561 hybrid_clock.cc:648] HybridClock initialized: now 1790606871507452 us; error 23 us; skew 500 ppm
Sep 28 14:47:51 dist-test-slave-j968 krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606871, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:51.510032 21561 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:47:51.510509 21561 webserver.cc:460] Webserver started at http://127.18.96.131:38457/ using document root <none> and password file <none>
I20260928 14:47:51.510780 21561 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:51.510848 21561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:51.511008 21561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:51.512127 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "a037d1ffcf114d369faa11e7b08fcb5d"
format_stamp: "Formatted at 2026-09-28 14:47:51 on dist-test-slave-j968"
server_key: "5e6b06c283a987b1b0674d9083d51c61"
server_key_iv: "8231f1c0e0770272cb684551551590e7"
server_key_version: "encryptionkey@0"
I20260928 14:47:51.512475 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "a037d1ffcf114d369faa11e7b08fcb5d"
format_stamp: "Formatted at 2026-09-28 14:47:51 on dist-test-slave-j968"
server_key: "5e6b06c283a987b1b0674d9083d51c61"
server_key_iv: "8231f1c0e0770272cb684551551590e7"
server_key_version: "encryptionkey@0"
I20260928 14:47:51.513782 21561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:47:51.514513 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.514681 21561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:51.514802 21561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "a037d1ffcf114d369faa11e7b08fcb5d"
format_stamp: "Formatted at 2026-09-28 14:47:51 on dist-test-slave-j968"
server_key: "5e6b06c283a987b1b0674d9083d51c61"
server_key_iv: "8231f1c0e0770272cb684551551590e7"
server_key_version: "encryptionkey@0"
I20260928 14:47:51.514876 21561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:51.535254 21561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:51.537410 21561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:51.537580 21561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:51.537832 21561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:51.538178 21561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:51.538234 21561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.538277 21561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:51.538302 21561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:51.544180 21561 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:46571
I20260928 14:47:51.544224 21689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:46571 every 8 connection(s)
I20260928 14:47:51.544538 21561 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:47:51.549255 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 21561
I20260928 14:47:51.549368 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:47:51.549561 18818 external_mini_cluster.cc:1428] Setting key 74412ce8a983ad9b9a4d67baa9ff364b
Sep 28 14:47:51 dist-test-slave-j968 krb5kdc[21186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606871, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:51.553721 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:51.545322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:60733 (local address 127.18.96.190:35489)
0928 14:47:51.545395 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:51.545397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:51.545612 (+ 215us) server_negotiation.cc:407] Connection header received
0928 14:47:51.546086 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:51.546087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:51.546097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:51.546151 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:51.547275 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.547787 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.548739 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.548926 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.548987 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:51.551305 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:51.551306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:51.551306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:51.551313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:51.552581 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:51.552922 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:51.552923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:51.552923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:51.553013 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:51.553228 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:51.553228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:51.553228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:51.553328 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:51.553376 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:51.553599 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 14:47:51.553659 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:47:51.553959 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:51.545273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35489 (local address 127.18.96.131:60733)
0928 14:47:51.545457 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:51.545493 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:51.546000 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:51.546204 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:51.546208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:51.546893 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:51.547153 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:51.547159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.547873 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.547874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:51.548666 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:51.548668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:51.548999 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:51.548999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:51.549129 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:51.549688 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:51.549697 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:51.551225 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:51.552647 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:51.552652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:51.552654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:51.552857 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:51.553078 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:51.553078 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:51.553079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:51.553173 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:51.553380 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:51.553384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:51.553495 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 14:47:51.553574 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:47:51.553691 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260928 14:47:51.554682 21690 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35489
I20260928 14:47:51.554781 21690 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:51.554986 21690 heartbeater.cc:511] Master 127.18.96.190:35489 requested a full tablet report, sending...
I20260928 14:47:51.555349 21231 ts_manager.cc:194] Registered new tserver with Master: a037d1ffcf114d369faa11e7b08fcb5d (127.18.96.131:46571)
I20260928 14:47:51.555718 21231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:60733
I20260928 14:47:51.561251 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:47:51.572304 21271 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:47:51.571316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.18.96.190:35489)
0928 14:47:51.571382 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:51.571384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:51.571525 (+ 141us) server_negotiation.cc:407] Connection header received
0928 14:47:51.571568 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:51.571568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:51.571578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:51.571629 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:51.572238 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35610: BlockingRecv error: recv got EOF from 127.0.0.1:35610 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:47:51.573191 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 21293
I20260928 14:47:51.577638 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 21427
I20260928 14:47:51.581805 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 21561
I20260928 14:47:51.586200 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 21199
2026-09-28T14:47:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1907 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:47:51 dist-test-slave-j968 krb5kdc[21702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:47:51 dist-test-slave-j968 krb5kdc[21702](info): set up 2 sockets
Sep 28 14:47:51 dist-test-slave-j968 krb5kdc[21702](info): commencing operation
krb5kdc: starting...
W20260928 14:47:52.636665 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:47:52 dist-test-slave-j968 krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:47:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:47:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:47:52.759549 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:39413
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:39987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:39413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:52.840421 21715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:52.840674 21715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:52.840710 21715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:52.841882 21715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:47:52.841954 21715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:52.841987 21715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:52.842015 21715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:47:52.842043 21715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:47:52.843530 21715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:39987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:39413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:39413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:52.843801 21715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:52.844060 21715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:52.848840 21723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:52.848944 21720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:52.849028 21715 server_base.cc:1034] running on GCE node
W20260928 14:47:52.849063 21721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:52.849309 21715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:52.849548 21715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:52.850684 21715 hybrid_clock.cc:648] HybridClock initialized: now 1790606872850675 us; error 43 us; skew 500 ppm
Sep 28 14:47:52 dist-test-slave-j968 krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606872, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:52.853184 21715 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:47:52.853649 21715 webserver.cc:460] Webserver started at http://127.18.96.190:40371/ using document root <none> and password file <none>
I20260928 14:47:52.853888 21715 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:52.853931 21715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:52.854105 21715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:52.855199 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9"
format_stamp: "Formatted at 2026-09-28 14:47:52 on dist-test-slave-j968"
server_key: "5370696c982c539312769e3e82f9d9f1"
server_key_iv: "9a2b730fd1aee83a69e932ffef569471"
server_key_version: "encryptionkey@0"
I20260928 14:47:52.855557 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9"
format_stamp: "Formatted at 2026-09-28 14:47:52 on dist-test-slave-j968"
server_key: "5370696c982c539312769e3e82f9d9f1"
server_key_iv: "9a2b730fd1aee83a69e932ffef569471"
server_key_version: "encryptionkey@0"
I20260928 14:47:52.857041 21715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:47:52.857750 21730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:52.857929 21715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:52.858062 21715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9"
format_stamp: "Formatted at 2026-09-28 14:47:52 on dist-test-slave-j968"
server_key: "5370696c982c539312769e3e82f9d9f1"
server_key_iv: "9a2b730fd1aee83a69e932ffef569471"
server_key_version: "encryptionkey@0"
I20260928 14:47:52.858141 21715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:52.878082 21715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:52.879958 21715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:52.880146 21715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:52.884167 21715 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:39413
I20260928 14:47:52.884238 21782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:39413 every 8 connection(s)
I20260928 14:47:52.884732 21715 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:47:52.885397 21783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:52.887639 21783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9: Bootstrap starting.
I20260928 14:47:52.888254 21783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:52.888554 21783 log.cc:824] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:52.889230 21783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9: No bootstrap required, opened a new log
I20260928 14:47:52.890769 21783 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39413 } }
I20260928 14:47:52.890908 21783 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:52.890959 21783 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf6f6ee2bca4d6bb5e4026ebc1a69d9, State: Initialized, Role: FOLLOWER
I20260928 14:47:52.891126 21783 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39413 } }
I20260928 14:47:52.891191 21783 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:47:52.891212 21783 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:47:52.891265 21783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:52.892221 21783 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39413 } }
I20260928 14:47:52.892307 21783 leader_election.cc:302] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcf6f6ee2bca4d6bb5e4026ebc1a69d9; no voters:
I20260928 14:47:52.892493 21783 leader_election.cc:288] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:47:52.892549 21786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:52.892745 21786 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [term 1 LEADER]: Becoming Leader. State: Replica: bcf6f6ee2bca4d6bb5e4026ebc1a69d9, State: Running, Role: LEADER
I20260928 14:47:52.892756 21783 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:47:52.892848 21786 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39413 } }
I20260928 14:47:52.893252 21787 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39413 } } }
I20260928 14:47:52.893260 21788 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcf6f6ee2bca4d6bb5e4026ebc1a69d9. Latest consensus state: current_term: 1 leader_uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf6f6ee2bca4d6bb5e4026ebc1a69d9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39413 } } }
I20260928 14:47:52.893463 21788 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:52.893430 21787 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:52.893839 21795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:47:52.894369 21795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:47:52.894505 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 21715
I20260928 14:47:52.894605 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:47:52.894785 18818 external_mini_cluster.cc:1428] Setting key 795a4346b20679b9385cb414a8d3f3db
I20260928 14:47:52.896003 21795 catalog_manager.cc:1347] Generated new cluster ID: ca5f178500414b838754efffcaf0b05a
I20260928 14:47:52.896057 21795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:47:52 dist-test-slave-j968 krb5kdc[21702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:52.903067 21795 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:47:52.903780 21795 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:47:52.904546 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:52.895237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.18.96.190:39413)
0928 14:47:52.895420 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:52.895430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:52.895532 (+ 102us) server_negotiation.cc:407] Connection header received
0928 14:47:52.895652 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:52.895657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:52.895701 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:52.895845 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:52.896554 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:52.897461 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:52.898215 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:52.898673 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:52.898791 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:52.901150 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:52.901159 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:52.901159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:52.901170 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:52.902820 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:52.903265 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:52.903267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:52.903268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:52.903353 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:52.903567 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:52.903568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:52.903569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:52.903789 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:52.903987 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:52.904127 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 14:47:52.904387 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260928 14:47:52.905397 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:47:52.910329 21795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcf6f6ee2bca4d6bb5e4026ebc1a69d9: Generated new TSK 0
I20260928 14:47:52.910516 21795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:47:52.924695 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:52.920648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.18.96.190:39413)
0928 14:47:52.920716 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:52.920718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:52.920818 (+ 100us) server_negotiation.cc:407] Connection header received
0928 14:47:52.920893 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:52.920895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:52.920907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:52.920967 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:52.921345 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:52.922182 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:52.922932 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:52.923302 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:52.923374 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:52.924140 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:52.924142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:52.924142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:47:52.924404 (+ 262us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:52.924425 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:47:52.924432 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:52.924497 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:52.924571 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 14:47:52.924606 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:47:52.927438 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 21715
2026-09-28T14:47:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1338 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:47:52 dist-test-slave-j968 krb5kdc[21817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:47:52 dist-test-slave-j968 krb5kdc[21817](info): set up 2 sockets
Sep 28 14:47:52 dist-test-slave-j968 krb5kdc[21817](info): commencing operation
krb5kdc: starting...
W20260928 14:47:53.956661 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:47:54 dist-test-slave-j968 krb5kdc[21817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:47:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:47:54.079274 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:40685
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:40685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:54.161752 21830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:54.162079 21830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:54.162106 21830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:54.163304 21830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:47:54.163347 21830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:54.163362 21830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:54.163376 21830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:47:54.163389 21830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:47:54.164935 21830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:40685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:40685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:54.165136 21830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:54.165302 21830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:54.170240 21835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:54.170269 21838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:54.170392 21830 server_base.cc:1034] running on GCE node
W20260928 14:47:54.170265 21836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:54.170804 21830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:54.171136 21830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:54.172277 21830 hybrid_clock.cc:648] HybridClock initialized: now 1790606874172255 us; error 36 us; skew 500 ppm
Sep 28 14:47:54 dist-test-slave-j968 krb5kdc[21817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606874, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:54.174790 21830 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:47:54.175316 21830 webserver.cc:460] Webserver started at http://127.18.96.190:38597/ using document root <none> and password file <none>
I20260928 14:47:54.175551 21830 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:54.175595 21830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:54.175745 21830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:54.176823 21830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "2c8dc8f24a29454890184760ce698808"
format_stamp: "Formatted at 2026-09-28 14:47:54 on dist-test-slave-j968"
server_key: "934bb317162a3eb1cb0e80caf7da9886"
server_key_iv: "cc355b1435cfcff3df5d57e47a726fcf"
server_key_version: "encryptionkey@0"
I20260928 14:47:54.177186 21830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "2c8dc8f24a29454890184760ce698808"
format_stamp: "Formatted at 2026-09-28 14:47:54 on dist-test-slave-j968"
server_key: "934bb317162a3eb1cb0e80caf7da9886"
server_key_iv: "cc355b1435cfcff3df5d57e47a726fcf"
server_key_version: "encryptionkey@0"
I20260928 14:47:54.178481 21830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:47:54.179201 21845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:54.179390 21830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:54.179488 21830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "2c8dc8f24a29454890184760ce698808"
format_stamp: "Formatted at 2026-09-28 14:47:54 on dist-test-slave-j968"
server_key: "934bb317162a3eb1cb0e80caf7da9886"
server_key_iv: "cc355b1435cfcff3df5d57e47a726fcf"
server_key_version: "encryptionkey@0"
I20260928 14:47:54.179569 21830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:54.202152 21830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:54.204260 21830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:54.204420 21830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:54.208585 21830 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:40685
I20260928 14:47:54.208626 21897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:40685 every 8 connection(s)
I20260928 14:47:54.209016 21830 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:47:54.210000 21898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:54.212368 21898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808: Bootstrap starting.
I20260928 14:47:54.212903 21898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:54.213140 21898 log.cc:824] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:54.213796 21898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808: No bootstrap required, opened a new log
I20260928 14:47:54.214072 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 21830
I20260928 14:47:54.214176 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:47:54.214354 18818 external_mini_cluster.cc:1428] Setting key b961993d3c00149be124aae0ddf0b2ac
I20260928 14:47:54.215924 21898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8dc8f24a29454890184760ce698808" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40685 } }
I20260928 14:47:54.216099 21898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:54.216140 21898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8dc8f24a29454890184760ce698808, State: Initialized, Role: FOLLOWER
I20260928 14:47:54.216279 21898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8dc8f24a29454890184760ce698808" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40685 } }
I20260928 14:47:54.216367 21898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:47:54.216404 21898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:47:54.216459 21898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:54.217530 21898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8dc8f24a29454890184760ce698808" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40685 } }
I20260928 14:47:54.217629 21898 leader_election.cc:302] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c8dc8f24a29454890184760ce698808; no voters:
I20260928 14:47:54.217823 21898 leader_election.cc:288] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:47:54.217844 21903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:54.217952 21903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [term 1 LEADER]: Becoming Leader. State: Replica: 2c8dc8f24a29454890184760ce698808, State: Running, Role: LEADER
I20260928 14:47:54.218034 21903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8dc8f24a29454890184760ce698808" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40685 } }
I20260928 14:47:54.218096 21898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:47:54.218374 21905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c8dc8f24a29454890184760ce698808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8dc8f24a29454890184760ce698808" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40685 } } }
I20260928 14:47:54.218465 21905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:54.218647 21904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c8dc8f24a29454890184760ce698808. Latest consensus state: current_term: 1 leader_uuid: "2c8dc8f24a29454890184760ce698808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8dc8f24a29454890184760ce698808" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40685 } } }
I20260928 14:47:54.218726 21904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808 [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:54.219058 21908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:47:54.219601 21908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:47:54.221133 21908 catalog_manager.cc:1347] Generated new cluster ID: 1a1ed41571a545738e0635062453f23c
I20260928 14:47:54.221186 21908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:47:54 dist-test-slave-j968 krb5kdc[21817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:54.230019 21908 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:47:54.230101 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:54.214858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.18.96.190:40685)
0928 14:47:54.215080 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:54.215087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:54.215112 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:47:54.215856 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:54.215899 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:54.215962 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:54.216128 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:54.216960 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:54.218025 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:54.219086 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:54.219760 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:54.219886 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:54.226179 (+ 6293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:54.226185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:54.226185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:54.226199 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:54.228303 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:54.228714 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:54.228717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:54.228717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:54.228827 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:54.229059 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:54.229060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:54.229060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:54.229378 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:54.229534 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:54.229690 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:47:54.229838 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":106,"threads_started":1}
I20260928 14:47:54.230804 21908 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:47:54.233493 21908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c8dc8f24a29454890184760ce698808: Generated new TSK 0
I20260928 14:47:54.233655 21908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:47:54.263278 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40685
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:54.346131 21924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:54.346323 21924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:54.346346 21924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:54.347533 21924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:54.347572 21924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:54.347599 21924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:47:54.349066 21924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:54.349299 21924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:54.349535 21924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:54.354472 21930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:54.354518 21929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:54.354507 21932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:54.354703 21924 server_base.cc:1034] running on GCE node
I20260928 14:47:54.354950 21924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:54.355224 21924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:54.356384 21924 hybrid_clock.cc:648] HybridClock initialized: now 1790606874356384 us; error 43 us; skew 500 ppm
Sep 28 14:47:54 dist-test-slave-j968 krb5kdc[21817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606874, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:54.358803 21924 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:47:54.359237 21924 webserver.cc:460] Webserver started at http://127.18.96.129:34941/ using document root <none> and password file <none>
I20260928 14:47:54.359462 21924 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:54.359527 21924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:54.359680 21924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:54.360771 21924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "34dee48464064794a0a7e924824ab881"
format_stamp: "Formatted at 2026-09-28 14:47:54 on dist-test-slave-j968"
server_key: "92a8b9fcfe7aa69d6bc7c8e44266e6aa"
server_key_iv: "a6e35e127e2fec586eba0fd21fce5060"
server_key_version: "encryptionkey@0"
I20260928 14:47:54.361136 21924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "34dee48464064794a0a7e924824ab881"
format_stamp: "Formatted at 2026-09-28 14:47:54 on dist-test-slave-j968"
server_key: "92a8b9fcfe7aa69d6bc7c8e44266e6aa"
server_key_iv: "a6e35e127e2fec586eba0fd21fce5060"
server_key_version: "encryptionkey@0"
I20260928 14:47:54.362464 21924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:47:54.363162 21939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:54.363339 21924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:54.363436 21924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "34dee48464064794a0a7e924824ab881"
format_stamp: "Formatted at 2026-09-28 14:47:54 on dist-test-slave-j968"
server_key: "92a8b9fcfe7aa69d6bc7c8e44266e6aa"
server_key_iv: "a6e35e127e2fec586eba0fd21fce5060"
server_key_version: "encryptionkey@0"
I20260928 14:47:54.363520 21924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:54.400496 21924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:54.402738 21924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:54.402913 21924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:54.403214 21924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:54.403568 21924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:54.403625 21924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:54.403661 21924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:54.403690 21924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:54.410068 21924 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:36781
I20260928 14:47:54.410161 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:36781 every 8 connection(s)
I20260928 14:47:54.410558 21924 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:47:54 dist-test-slave-j968 krb5kdc[21817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606874, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:54.418584 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 21924
I20260928 14:47:54.418706 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:47:54.418931 18818 external_mini_cluster.cc:1428] Setting key b88293d6d4508cb741ede2ce684ccc80
I20260928 14:47:54.420567 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:54.411340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:42731 (local address 127.18.96.190:40685)
0928 14:47:54.411390 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:54.411393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:54.411594 (+ 201us) server_negotiation.cc:407] Connection header received
0928 14:47:54.412203 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:54.412206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:54.412218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:54.412272 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:54.413250 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:54.413762 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:54.414786 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:54.415021 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:54.415108 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:54.417296 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:54.417298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:54.417298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:54.417306 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:54.418700 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:54.419152 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:54.419153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:54.419154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:54.419224 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:54.419530 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:54.419531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:54.419532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:54.419681 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:54.419750 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:54.420221 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 14:47:54.420387 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:47:54.420892 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:54.411296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:40685 (local address 127.18.96.129:42731)
0928 14:47:54.411438 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:54.411499 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:54.412094 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:54.412344 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:54.412347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:54.412952 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:54.413147 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:54.413152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:54.413865 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:54.413866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:54.414689 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:54.414691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:54.415120 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:54.415121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:54.415256 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:54.415750 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:54.415758 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:54.417201 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:54.418799 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:54.418806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:54.418808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:54.419071 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:54.419306 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:54.419307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:54.419307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:54.419441 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:54.419818 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:54.419825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:54.420025 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 14:47:54.420164 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 14:47:54.420363 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260928 14:47:54.421460 22053 heartbeater.cc:347] Connected to a master server at 127.18.96.190:40685
I20260928 14:47:54.421553 22053 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:54.421756 22053 heartbeater.cc:511] Master 127.18.96.190:40685 requested a full tablet report, sending...
I20260928 14:47:54.422411 21862 ts_manager.cc:194] Registered new tserver with Master: 34dee48464064794a0a7e924824ab881 (127.18.96.129:36781)
I20260928 14:47:54.423046 21862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:42731
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
I20260928 14:47:55.424208 22053 heartbeater.cc:503] Master 127.18.96.190:40685 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
W20260928 14:47:55.925148 18818 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.96.130: real 1.505s user 0.000s sys 0.003s
I20260928 14:47:55.925510 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40685
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:56.008649 22058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:56.008862 22058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:56.008885 22058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:56.010164 22058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:56.010205 22058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:56.010232 22058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:47:56.011706 22058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:56.011991 22058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:56.012254 22058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:56.017565 22063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:56.017647 22066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:56.017745 22058 server_base.cc:1034] running on GCE node
W20260928 14:47:56.017571 22064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:56.018091 22058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:56.018389 22058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:56.019549 22058 hybrid_clock.cc:648] HybridClock initialized: now 1790606876019528 us; error 38 us; skew 500 ppm
Sep 28 14:47:56 dist-test-slave-j968 krb5kdc[21817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:56.026104 22058 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:47:56.026544 22058 webserver.cc:460] Webserver started at http://127.18.96.130:33965/ using document root <none> and password file <none>
I20260928 14:47:56.026788 22058 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:56.026857 22058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:56.027009 22058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:56.028146 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "88e7f2cfd0b14ecca15046045f9c6104"
format_stamp: "Formatted at 2026-09-28 14:47:56 on dist-test-slave-j968"
server_key: "f32ec514ba291bd43f37eef2341b6957"
server_key_iv: "e85d40e2b0d4bf393847cbed490a82a4"
server_key_version: "encryptionkey@0"
I20260928 14:47:56.028498 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "88e7f2cfd0b14ecca15046045f9c6104"
format_stamp: "Formatted at 2026-09-28 14:47:56 on dist-test-slave-j968"
server_key: "f32ec514ba291bd43f37eef2341b6957"
server_key_iv: "e85d40e2b0d4bf393847cbed490a82a4"
server_key_version: "encryptionkey@0"
I20260928 14:47:56.029827 22058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:56.030624 22073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:56.030809 22058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:47:56.030915 22058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "88e7f2cfd0b14ecca15046045f9c6104"
format_stamp: "Formatted at 2026-09-28 14:47:56 on dist-test-slave-j968"
server_key: "f32ec514ba291bd43f37eef2341b6957"
server_key_iv: "e85d40e2b0d4bf393847cbed490a82a4"
server_key_version: "encryptionkey@0"
I20260928 14:47:56.030988 22058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:56.057420 22058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:56.059484 22058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:56.059621 22058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:56.059813 22058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:56.060163 22058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:56.060197 22058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:56.060220 22058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:56.060233 22058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:56.067358 22058 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:35985
I20260928 14:47:56.067420 22186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:35985 every 8 connection(s)
I20260928 14:47:56.067855 22058 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:47:56.070930 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 22058
I20260928 14:47:56.071046 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:47:56.071244 18818 external_mini_cluster.cc:1428] Setting key d904ef3e900331fe151dc4d81e31437d
Sep 28 14:47:56 dist-test-slave-j968 krb5kdc[21817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606876, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:56.079258 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:56.068737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:49007 (local address 127.18.96.190:40685)
0928 14:47:56.068902 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:56.068905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:56.068984 (+ 79us) server_negotiation.cc:407] Connection header received
0928 14:47:56.069502 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:56.069503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:56.069515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:56.069589 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:56.070497 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:56.071349 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:56.072647 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:56.073031 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:56.073107 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:56.075727 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:56.075729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:56.075729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:56.075737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:56.077471 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:56.078229 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:56.078231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:56.078231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:56.078301 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:56.078568 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:56.078569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:56.078569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:56.078695 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:56.078768 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:56.079047 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 14:47:56.079136 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
I20260928 14:47:56.079468 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:56.068699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:40685 (local address 127.18.96.130:49007)
0928 14:47:56.068862 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:56.068912 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:56.069405 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:56.069642 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:56.069645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:56.070210 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:56.070406 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:56.070411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:56.071569 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:56.071571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:56.072550 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:56.072553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:56.073122 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:56.073123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:56.073228 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:56.073795 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:56.073806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:56.075586 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:56.077579 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:56.077586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:56.077589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:56.078117 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:56.078374 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:56.078383 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:56.078383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:56.078491 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:56.078773 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:56.078779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:56.078918 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 14:47:56.079026 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:47:56.079203 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
I20260928 14:47:56.080062 22187 heartbeater.cc:347] Connected to a master server at 127.18.96.190:40685
I20260928 14:47:56.080168 22187 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:56.080339 22187 heartbeater.cc:511] Master 127.18.96.190:40685 requested a full tablet report, sending...
I20260928 14:47:56.080731 21862 ts_manager.cc:194] Registered new tserver with Master: 88e7f2cfd0b14ecca15046045f9c6104 (127.18.96.130:35985)
I20260928 14:47:56.081115 21862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:49007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
I20260928 14:47:57.082034 22187 heartbeater.cc:503] Master 127.18.96.190:40685 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
W20260928 14:47:57.386886 18818 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.96.131: real 1.315s user 0.000s sys 0.003s
I20260928 14:47:57.387308 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40685
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:57.475193 22193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:57.475445 22193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:57.475481 22193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:57.476709 22193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:57.476771 22193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:57.476814 22193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:47:57.478430 22193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:57.478706 22193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:57.478971 22193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:57.484372 22198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:57.484382 22199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:57.484511 22193 server_base.cc:1034] running on GCE node
W20260928 14:47:57.484382 22201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:57.484864 22193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:57.485076 22193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:57.486254 22193 hybrid_clock.cc:648] HybridClock initialized: now 1790606877486240 us; error 27 us; skew 500 ppm
Sep 28 14:47:57 dist-test-slave-j968 krb5kdc[21817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:57.488921 22193 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:47:57.489434 22193 webserver.cc:460] Webserver started at http://127.18.96.131:43485/ using document root <none> and password file <none>
I20260928 14:47:57.489676 22193 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:57.489743 22193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:57.489903 22193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:57.491086 22193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "427730500111434cb184b7daf79a08a5"
format_stamp: "Formatted at 2026-09-28 14:47:57 on dist-test-slave-j968"
server_key: "f7e37bac8a8bc67e78a3fffa95bc5140"
server_key_iv: "25e5fda79264c423f73f5676d8a79f22"
server_key_version: "encryptionkey@0"
I20260928 14:47:57.491451 22193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "427730500111434cb184b7daf79a08a5"
format_stamp: "Formatted at 2026-09-28 14:47:57 on dist-test-slave-j968"
server_key: "f7e37bac8a8bc67e78a3fffa95bc5140"
server_key_iv: "25e5fda79264c423f73f5676d8a79f22"
server_key_version: "encryptionkey@0"
I20260928 14:47:57.558913 22193 fs_manager.cc:691] Time spent creating directory manager: real 0.067s user 0.001s sys 0.003s
I20260928 14:47:57.560041 22208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:57.560266 22193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:57.560375 22193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "427730500111434cb184b7daf79a08a5"
format_stamp: "Formatted at 2026-09-28 14:47:57 on dist-test-slave-j968"
server_key: "f7e37bac8a8bc67e78a3fffa95bc5140"
server_key_iv: "25e5fda79264c423f73f5676d8a79f22"
server_key_version: "encryptionkey@0"
I20260928 14:47:57.560447 22193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:57.582768 22193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:57.584949 22193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:57.585119 22193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:57.585330 22193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:57.585690 22193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:57.585726 22193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:57.585749 22193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:57.585793 22193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:57.591919 22193 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:34397
I20260928 14:47:57.591998 22321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:34397 every 8 connection(s)
I20260928 14:47:57.592294 22193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:47:57.594130 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 22193
I20260928 14:47:57.594245 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:47:57.594441 18818 external_mini_cluster.cc:1428] Setting key ddc95186a0a1ec545289d5d0bf967b6a
Sep 28 14:47:57 dist-test-slave-j968 krb5kdc[21817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606877, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:57.602435 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:57.593126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:33241 (local address 127.18.96.190:40685)
0928 14:47:57.593248 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:57.593250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:57.593427 (+ 177us) server_negotiation.cc:407] Connection header received
0928 14:47:57.593961 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:57.593963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:57.593980 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:57.594076 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:57.595357 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:57.596047 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:57.597112 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:57.597305 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:57.597352 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:57.599753 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:57.599755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:57.599755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:57.599762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:57.601235 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:57.601630 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:57.601631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:57.601631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:57.601685 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:57.601879 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:57.601879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:57.601880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:57.601984 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:57.602033 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:57.602295 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 14:47:57.602361 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260928 14:47:57.602762 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:57.593120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:40685 (local address 127.18.96.131:33241)
0928 14:47:57.593290 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:57.593333 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:57.593819 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:57.594146 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:57.594149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:57.594965 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:57.595280 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:57.595286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:57.596154 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:57.596155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:57.597030 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:57.597032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:57.597374 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:57.597375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:57.597511 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:57.598117 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:57.598125 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:57.599653 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:57.601317 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:57.601323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:57.601325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:57.601558 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:57.601729 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:57.601729 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:57.601729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:57.601818 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:57.602042 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:57.602047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:57.602163 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 14:47:57.602249 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:47:57.602476 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260928 14:47:57.603314 22322 heartbeater.cc:347] Connected to a master server at 127.18.96.190:40685
I20260928 14:47:57.603415 22322 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:57.603628 22322 heartbeater.cc:511] Master 127.18.96.190:40685 requested a full tablet report, sending...
I20260928 14:47:57.604027 21862 ts_manager.cc:194] Registered new tserver with Master: 427730500111434cb184b7daf79a08a5 (127.18.96.131:34397)
I20260928 14:47:57.604400 21862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:33241
I20260928 14:47:57.606184 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:47:57 dist-test-slave-j968 krb5kdc[21817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606877, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 14:47:57 dist-test-slave-j968 krb5kdc[21817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606877, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:57.713881 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:57.705105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.18.96.190:40685)
0928 14:47:57.705173 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:57.705175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:57.705195 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:47:57.705246 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:57.705247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:57.705256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:57.705358 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:57.706069 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:57.706581 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:57.707352 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:57.707559 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:57.707637 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:57.710693 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:57.710695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:57.710695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:57.710702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:57.712288 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:57.712728 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:57.712730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:57.712730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:57.712795 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:57.713219 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:57.713220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:57.713221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:57.713407 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:57.713542 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:57.713661 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 14:47:57.713763 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 14:47:57.714193 21862 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33114
I20260928 14:47:57.714783 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 21924
I20260928 14:47:57.719650 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 22058
I20260928 14:47:57.724472 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 22193
I20260928 14:47:57.728524 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 21830
2026-09-28T14:47:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4803 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:47:58 dist-test-slave-j968 krb5kdc[22335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:47:58 dist-test-slave-j968 krb5kdc[22335](info): set up 2 sockets
Sep 28 14:47:58 dist-test-slave-j968 krb5kdc[22335](info): commencing operation
krb5kdc: starting...
W20260928 14:47:59.244791 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.496s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:47:59.366787 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:36789
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:36789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:59.447880 22348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:59.448087 22348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:59.448110 22348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:59.449311 22348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:47:59.449352 22348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:59.449366 22348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:59.449378 22348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:47:59.449433 22348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:47:59.450927 22348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:36789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:36789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:59.451179 22348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:59.451390 22348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:59.456777 22354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:59.456801 22356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:59.456809 22353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:59.456892 22348 server_base.cc:1034] running on GCE node
I20260928 14:47:59.457145 22348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:59.457435 22348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:59.458627 22348 hybrid_clock.cc:648] HybridClock initialized: now 1790606879458618 us; error 39 us; skew 500 ppm
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:59.461467 22348 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:47:59.461871 22348 webserver.cc:460] Webserver started at http://127.18.96.190:34821/ using document root <none> and password file <none>
I20260928 14:47:59.462131 22348 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:59.462176 22348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:59.462307 22348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:59.463402 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "773edc6535c2419b9cda6f9ce56336e9"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "9ffc870e2eb64c00fb4dff543ec9ec4a"
server_key_iv: "91c97154fd133bf0fb4756c976bfd606"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.463763 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "773edc6535c2419b9cda6f9ce56336e9"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "9ffc870e2eb64c00fb4dff543ec9ec4a"
server_key_iv: "91c97154fd133bf0fb4756c976bfd606"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.465305 22348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:59.466071 22363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.466317 22348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:47:59.466411 22348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "773edc6535c2419b9cda6f9ce56336e9"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "9ffc870e2eb64c00fb4dff543ec9ec4a"
server_key_iv: "91c97154fd133bf0fb4756c976bfd606"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.466493 22348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:59.486474 22348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:59.489002 22348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:59.489192 22348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:59.493290 22348 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:36789
I20260928 14:47:59.493330 22415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:36789 every 8 connection(s)
I20260928 14:47:59.493827 22348 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:47:59.494604 22416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:59.496851 22416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9: Bootstrap starting.
I20260928 14:47:59.497459 22416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:59.497695 22416 log.cc:824] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:59.498310 22416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9: No bootstrap required, opened a new log
I20260928 14:47:59.499923 22416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773edc6535c2419b9cda6f9ce56336e9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36789 } }
I20260928 14:47:59.500078 22416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:59.500102 22416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773edc6535c2419b9cda6f9ce56336e9, State: Initialized, Role: FOLLOWER
I20260928 14:47:59.500231 22416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773edc6535c2419b9cda6f9ce56336e9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36789 } }
I20260928 14:47:59.500305 22416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:47:59.500327 22416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:47:59.500355 22416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:59.501225 22416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773edc6535c2419b9cda6f9ce56336e9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36789 } }
I20260928 14:47:59.501309 22416 leader_election.cc:302] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 773edc6535c2419b9cda6f9ce56336e9; no voters:
I20260928 14:47:59.501427 22416 leader_election.cc:288] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:47:59.501494 22419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:59.501665 22419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [term 1 LEADER]: Becoming Leader. State: Replica: 773edc6535c2419b9cda6f9ce56336e9, State: Running, Role: LEADER
I20260928 14:47:59.501688 22416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:47:59.501754 22419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773edc6535c2419b9cda6f9ce56336e9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36789 } }
I20260928 14:47:59.501888 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 22348
I20260928 14:47:59.502012 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:47:59.502159 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 773edc6535c2419b9cda6f9ce56336e9. Latest consensus state: current_term: 1 leader_uuid: "773edc6535c2419b9cda6f9ce56336e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773edc6535c2419b9cda6f9ce56336e9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36789 } } }
I20260928 14:47:59.502219 18818 external_mini_cluster.cc:1428] Setting key b5d6ad24049c662ad167d57e14e3c660
I20260928 14:47:59.502312 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:59.502907 22420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "773edc6535c2419b9cda6f9ce56336e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773edc6535c2419b9cda6f9ce56336e9" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36789 } } }
I20260928 14:47:59.502974 22420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:59.503106 22429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:47:59.503676 22429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:47:59.505465 22429 catalog_manager.cc:1347] Generated new cluster ID: 4ff2796d192f4025a2cc2af96f79b00e
I20260928 14:47:59.505517 22429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:59.510448 22429 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:47:59.511121 22429 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:47:59.514410 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:59.502755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.18.96.190:36789)
0928 14:47:59.503179 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:59.503192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:59.503231 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:47:59.503359 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:59.503367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:59.503435 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:59.503614 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:59.504483 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.505963 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.507019 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.507675 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.507791 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:59.510687 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:59.510691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:59.510691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:59.510703 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:59.512674 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:59.513083 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:59.513086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:59.513086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:59.513186 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:59.513407 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:59.513407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:59.513408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:59.513685 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:59.513827 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:59.513960 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 14:47:59.514199 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":83,"threads_started":1}
I20260928 14:47:59.515691 22429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 773edc6535c2419b9cda6f9ce56336e9: Generated new TSK 0
I20260928 14:47:59.515861 22429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:47:59.546870 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36789
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:59.629156 22442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:59.629401 22442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:59.629441 22442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:59.630618 22442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:59.630687 22442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:59.630734 22442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:47:59.632367 22442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:59.632829 22442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:59.633113 22442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:59.638623 22447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:59.638646 22442 server_base.cc:1034] running on GCE node
W20260928 14:47:59.638623 22450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:59.638628 22448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:59.639165 22442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:59.639447 22442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:59.640663 22442 hybrid_clock.cc:648] HybridClock initialized: now 1790606879640639 us; error 38 us; skew 500 ppm
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:59.643563 22442 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:47:59.644064 22442 webserver.cc:460] Webserver started at http://127.18.96.129:45389/ using document root <none> and password file <none>
I20260928 14:47:59.644299 22442 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:59.644343 22442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:59.644497 22442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:59.645547 22442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "7bd4bfe57ca244fd9a6b48d23a906d70"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "95cce16e973e24b422f840daac79af8c"
server_key_iv: "66a651cc36ec4371debe68fc05a511e6"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.645915 22442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "7bd4bfe57ca244fd9a6b48d23a906d70"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "95cce16e973e24b422f840daac79af8c"
server_key_iv: "66a651cc36ec4371debe68fc05a511e6"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.647248 22442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:47:59.648039 22457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.648198 22442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:59.648288 22442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "7bd4bfe57ca244fd9a6b48d23a906d70"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "95cce16e973e24b422f840daac79af8c"
server_key_iv: "66a651cc36ec4371debe68fc05a511e6"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.648342 22442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:59.661589 22442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:59.664038 22442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:59.664152 22442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:59.664345 22442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:59.664670 22442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:59.664710 22442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.664733 22442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:59.664778 22442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.670670 22442 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:35251
I20260928 14:47:59.670733 22570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:35251 every 8 connection(s)
I20260928 14:47:59.671056 22442 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:47:59.672190 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 22442
I20260928 14:47:59.672274 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:47:59.672468 18818 external_mini_cluster.cc:1428] Setting key bfe6cb44bd140e9e08d26af0865385a6
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
I20260928 14:47:59.682116 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:59.671782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:51295 (local address 127.18.96.190:36789)
0928 14:47:59.671831 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:59.671834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:59.672063 (+ 229us) server_negotiation.cc:407] Connection header received
0928 14:47:59.673305 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:59.673306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:59.673319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:59.673365 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:59.674042 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.674807 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.676303 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.676498 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.676557 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:59.679112 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:59.679115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:59.679116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:59.679128 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:59.680581 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:59.681092 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:59.681093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:59.681094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:59.681152 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:59.681405 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:59.681406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:59.681406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:59.681528 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:59.681605 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:59.681956 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 14:47:59.682040 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:47:59.682364 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:59.671743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:36789 (local address 127.18.96.129:51295)
0928 14:47:59.671906 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:59.671958 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:59.673166 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:59.673448 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:59.673452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:59.673731 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:59.673926 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:59.673931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.674917 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.674918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:59.676213 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:59.676217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.676596 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.676598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:59.676763 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:59.677312 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:59.677324 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:59.679001 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:59.680704 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:59.680712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:59.680715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:59.681008 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:59.681219 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:59.681220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:59.681221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:59.681336 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:59.681596 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:59.681603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:59.681784 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 14:47:59.681920 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 14:47:59.682093 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260928 14:47:59.682969 22571 heartbeater.cc:347] Connected to a master server at 127.18.96.190:36789
I20260928 14:47:59.683085 22571 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:59.683338 22571 heartbeater.cc:511] Master 127.18.96.190:36789 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
I20260928 14:47:59.684007 22380 ts_manager.cc:194] Registered new tserver with Master: 7bd4bfe57ca244fd9a6b48d23a906d70 (127.18.96.129:35251)
I20260928 14:47:59.684638 22380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:51295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:47:59.698669 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36789
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:59.782480 22576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:59.782691 22576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:59.782713 22576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:59.783910 22576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:59.783949 22576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:59.783975 22576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:47:59.785392 22576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:59.785617 22576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:59.785836 22576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:59.791005 22582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:59.791185 22576 server_base.cc:1034] running on GCE node
W20260928 14:47:59.791081 22581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:59.791364 22584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:59.791611 22576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:59.791903 22576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:59.793064 22576 hybrid_clock.cc:648] HybridClock initialized: now 1790606879793046 us; error 32 us; skew 500 ppm
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:59.795694 22576 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:47:59.796226 22576 webserver.cc:460] Webserver started at http://127.18.96.130:46649/ using document root <none> and password file <none>
I20260928 14:47:59.796455 22576 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:59.796532 22576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:59.796689 22576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:59.797724 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "10320e4d61b04f4ba6706523dcc0cb46"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "30aec66a16a302e708e4fd005d7cfe2c"
server_key_iv: "0d1af7b6f5f6c46697cc49c15a2f1902"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.798082 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "10320e4d61b04f4ba6706523dcc0cb46"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "30aec66a16a302e708e4fd005d7cfe2c"
server_key_iv: "0d1af7b6f5f6c46697cc49c15a2f1902"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.799415 22576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:47:59.800208 22591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.800374 22576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.800473 22576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "10320e4d61b04f4ba6706523dcc0cb46"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "30aec66a16a302e708e4fd005d7cfe2c"
server_key_iv: "0d1af7b6f5f6c46697cc49c15a2f1902"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.800580 22576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:59.809746 22576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:59.811807 22576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:59.812002 22576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:59.812234 22576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:59.812544 22576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:59.812598 22576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.812638 22576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:59.812666 22576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.818384 22576 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:39835
I20260928 14:47:59.818425 22704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:39835 every 8 connection(s)
I20260928 14:47:59.818781 22576 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:47:59.824090 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 22576
I20260928 14:47:59.824214 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:47:59.824401 18818 external_mini_cluster.cc:1428] Setting key 1a84ec403c8928cd22ced72a7756d406
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:59.829495 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:59.819565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:53003 (local address 127.18.96.190:36789)
0928 14:47:59.819628 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:59.819630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:59.819827 (+ 197us) server_negotiation.cc:407] Connection header received
0928 14:47:59.820796 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:59.820797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:59.820807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:59.820867 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:59.821452 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.821948 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.822916 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.823142 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.823195 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:59.825727 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:59.825729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:59.825729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:59.825737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:59.827638 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:59.828288 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:59.828289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:59.828289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:59.828344 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:59.828673 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:59.828674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:59.828674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:59.828805 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:59.828883 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:59.829304 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 14:47:59.829421 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:47:59.829692 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:59.819522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:36789 (local address 127.18.96.130:53003)
0928 14:47:59.819696 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:59.819738 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:59.820701 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:59.820924 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:59.820927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:59.821184 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:59.821380 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:59.821384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.822029 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.822029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:59.822843 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:59.822845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.823216 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.823216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:59.823318 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:59.823916 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:59.823926 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:59.825539 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:59.827843 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:59.827853 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:59.827856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:59.828215 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:59.828411 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:59.828412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:59.828413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:59.828565 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:59.828886 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:59.828893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:59.829079 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 14:47:59.829199 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 14:47:59.829402 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260928 14:47:59.830297 22705 heartbeater.cc:347] Connected to a master server at 127.18.96.190:36789
I20260928 14:47:59.830430 22705 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:59.830637 22705 heartbeater.cc:511] Master 127.18.96.190:36789 requested a full tablet report, sending...
I20260928 14:47:59.831120 22380 ts_manager.cc:194] Registered new tserver with Master: 10320e4d61b04f4ba6706523dcc0cb46 (127.18.96.130:39835)
I20260928 14:47:59.831558 22380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:53003
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:47:59.851651 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36789
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:59.933195 22710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:59.933415 22710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:59.933439 22710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:59.934515 22710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:59.934551 22710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:59.934582 22710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:47:59.936043 22710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:47:59.936298 22710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:59.936564 22710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:59.941795 22715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:59.941787 22718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:59.942066 22710 server_base.cc:1034] running on GCE node
W20260928 14:47:59.942203 22716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:59.942489 22710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:59.942754 22710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:59.943939 22710 hybrid_clock.cc:648] HybridClock initialized: now 1790606879943920 us; error 27 us; skew 500 ppm
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:59.946648 22710 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:47:59.947131 22710 webserver.cc:460] Webserver started at http://127.18.96.131:35293/ using document root <none> and password file <none>
I20260928 14:47:59.947366 22710 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:59.947410 22710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:59.947543 22710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:47:59.948688 22710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "384ecd78d7984bd8be8c1ed8a04246e3"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "12e7ab5acd5105b08d5beff117fc650c"
server_key_iv: "a5a90c000364c0d436129de4cf6deba3"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.949052 22710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "384ecd78d7984bd8be8c1ed8a04246e3"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "12e7ab5acd5105b08d5beff117fc650c"
server_key_iv: "a5a90c000364c0d436129de4cf6deba3"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.950402 22710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:47:59.951174 22725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.951372 22710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:47:59.951455 22710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "384ecd78d7984bd8be8c1ed8a04246e3"
format_stamp: "Formatted at 2026-09-28 14:47:59 on dist-test-slave-j968"
server_key: "12e7ab5acd5105b08d5beff117fc650c"
server_key_iv: "a5a90c000364c0d436129de4cf6deba3"
server_key_version: "encryptionkey@0"
I20260928 14:47:59.951514 22710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:59.967365 22710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:59.969681 22710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:59.969841 22710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:59.970082 22710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:59.970430 22710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:59.970463 22710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.970487 22710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:59.970500 22710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:59.976174 22710 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:45925
I20260928 14:47:59.976308 22838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:45925 every 8 connection(s)
I20260928 14:47:59.976593 22710 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:47:59.976971 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 22710
I20260928 14:47:59.977097 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:47:59.977299 18818 external_mini_cluster.cc:1428] Setting key 38cd8170e77b2f9aa771c5db3dd64f26
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:47:59.986084 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:59.977323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:49735 (local address 127.18.96.190:36789)
0928 14:47:59.977371 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:59.977373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:59.977663 (+ 290us) server_negotiation.cc:407] Connection header received
0928 14:47:59.978790 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:59.978791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:59.978804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:59.978834 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:59.979491 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.980089 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.981059 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.981274 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.981359 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:59.983453 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:59.983454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:59.983455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:59.983462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:59.984739 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:59.985147 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:59.985148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:59.985149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:59.985205 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:59.985447 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:59.985448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:59.985448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:59.985562 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:59.985638 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:59.985906 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 14:47:59.985984 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:47:59.986253 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:59.977324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:36789 (local address 127.18.96.131:49735)
0928 14:47:59.977541 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:59.977597 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:59.978676 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:59.978940 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:59.978943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:59.979200 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:59.979399 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:59.979403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.980176 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.980178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:59.980968 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:59.980970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:59.981367 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:59.981368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:59.981492 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:59.982032 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:59.982040 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:59.983357 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:59.984828 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:59.984834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:59.984837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:59.985050 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:59.985272 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:59.985273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:59.985274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:59.985379 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:59.985629 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:59.985635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:59.985772 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 14:47:59.985873 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:47:59.986025 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":123,"threads_started":1}
I20260928 14:47:59.986688 22839 heartbeater.cc:347] Connected to a master server at 127.18.96.190:36789
I20260928 14:47:59.986780 22839 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:59.986917 22839 heartbeater.cc:511] Master 127.18.96.190:36789 requested a full tablet report, sending...
I20260928 14:47:59.987295 22380 ts_manager.cc:194] Registered new tserver with Master: 384ecd78d7984bd8be8c1ed8a04246e3 (127.18.96.131:45925)
I20260928 14:47:59.987664 22380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:49735
I20260928 14:47:59.988938 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:47:59 dist-test-slave-j968 krb5kdc[22335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:48:00 dist-test-slave-j968 krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:48:00.013198 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:00.005194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.18.96.129:35251)
0928 14:48:00.005385 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:00.005390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:00.005400 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:00.005427 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:00.005428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:00.005442 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:00.005511 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:00.006099 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:00.006836 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:00.007568 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:00.007957 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:00.008012 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:00.010351 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:00.010358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:00.010359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:00.010365 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:00.012019 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:00.012331 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:00.012335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:00.012335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:00.012411 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:00.012602 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:00.012602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:00.012603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:00.012738 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:00.012840 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:00.012978 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 14:48:00.013086 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":112,"threads_started":1}
Sep 28 14:48:00 dist-test-slave-j968 krb5kdc[22335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:00.021895 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:00.014637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.18.96.190:36789)
0928 14:48:00.014694 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:00.014695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:00.014704 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:00.014736 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:00.014737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:00.014744 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:00.014791 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:00.015448 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:00.016163 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:00.016869 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:00.017041 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:00.017102 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:00.019364 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:00.019365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:00.019366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:00.019371 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:00.020834 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:00.021198 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:00.021198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:00.021199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:00.021249 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:00.021458 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:00.021459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:00.021459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:00.021597 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:00.021668 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:00.021759 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:48:00.021818 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 14:48:00.022146 22380 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51900
I20260928 14:48:00.022493 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 22442
I20260928 14:48:00.027722 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 22576
I20260928 14:48:00.032361 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 22710
I20260928 14:48:00.037029 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 22348
2026-09-28T14:48:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2308 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:00 dist-test-slave-j968 krb5kdc[22851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:00 dist-test-slave-j968 krb5kdc[22851](info): set up 2 sockets
Sep 28 14:48:00 dist-test-slave-j968 krb5kdc[22851](info): commencing operation
krb5kdc: starting...
W20260928 14:48:01.080854 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:48:01 dist-test-slave-j968 krb5kdc[22851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:48:01.203702 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43455
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:43455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:01.295248 22864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:01.295482 22864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:01.295506 22864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:01.296705 22864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:01.296749 22864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:01.296764 22864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:01.296777 22864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:01.296789 22864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:01.298574 22864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:43455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:01.298867 22864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:01.299126 22864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:01.304751 22872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:01.304803 22869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:01.304759 22870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:01.305205 22864 server_base.cc:1034] running on GCE node
I20260928 14:48:01.305505 22864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:01.305827 22864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:01.307019 22864 hybrid_clock.cc:648] HybridClock initialized: now 1790606881307002 us; error 23 us; skew 500 ppm
Sep 28 14:48:01 dist-test-slave-j968 krb5kdc[22851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:01.309633 22864 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:48:01.310060 22864 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:01.310137 22864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:01.310290 22864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:01.311386 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "80183b8859dc47948eefd23ae9bc6910"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "78378c037775f6d0b21a6a5fa6c73561"
server_key_iv: "ef35526b459f357db0c1d31642298be5"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.311769 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "80183b8859dc47948eefd23ae9bc6910"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "78378c037775f6d0b21a6a5fa6c73561"
server_key_iv: "ef35526b459f357db0c1d31642298be5"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.313161 22864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:01.314085 22878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.314312 22864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:01.314437 22864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "80183b8859dc47948eefd23ae9bc6910"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "78378c037775f6d0b21a6a5fa6c73561"
server_key_iv: "ef35526b459f357db0c1d31642298be5"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.314517 22864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:01.327605 22864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:01.329886 22864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:01.330081 22864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:01.334187 22864 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:43455
I20260928 14:48:01.334208 22930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:43455 every 8 connection(s)
I20260928 14:48:01.334511 22864 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:01.335429 22931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:01.337826 22931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910: Bootstrap starting.
I20260928 14:48:01.338377 22931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:01.338690 22931 log.cc:824] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:01.339421 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 22864
I20260928 14:48:01.339488 22931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910: No bootstrap required, opened a new log
I20260928 14:48:01.339531 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:01.339731 18818 external_mini_cluster.cc:1428] Setting key 521da6295d5fdcfa983040758ced1f4b
I20260928 14:48:01.341473 22931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80183b8859dc47948eefd23ae9bc6910" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43455 } }
I20260928 14:48:01.341616 22931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:01.341640 22931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80183b8859dc47948eefd23ae9bc6910, State: Initialized, Role: FOLLOWER
I20260928 14:48:01.341825 22931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80183b8859dc47948eefd23ae9bc6910" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43455 } }
I20260928 14:48:01.341933 22931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:01.341966 22931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:01.342018 22931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:01.342711 22931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80183b8859dc47948eefd23ae9bc6910" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43455 } }
I20260928 14:48:01.342797 22931 leader_election.cc:302] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80183b8859dc47948eefd23ae9bc6910; no voters:
I20260928 14:48:01.343003 22931 leader_election.cc:288] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:01.343091 22936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:01.343228 22936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [term 1 LEADER]: Becoming Leader. State: Replica: 80183b8859dc47948eefd23ae9bc6910, State: Running, Role: LEADER
I20260928 14:48:01.343276 22931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:01.343312 22936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80183b8859dc47948eefd23ae9bc6910" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43455 } }
I20260928 14:48:01.343604 22937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80183b8859dc47948eefd23ae9bc6910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80183b8859dc47948eefd23ae9bc6910" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43455 } } }
I20260928 14:48:01.343694 22937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:01.343835 22938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80183b8859dc47948eefd23ae9bc6910. Latest consensus state: current_term: 1 leader_uuid: "80183b8859dc47948eefd23ae9bc6910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80183b8859dc47948eefd23ae9bc6910" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43455 } } }
I20260928 14:48:01.343951 22938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:01.344120 22942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:01.344651 22942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:01.347172 22942 catalog_manager.cc:1347] Generated new cluster ID: fedc9f97028840389c5edd9a37aacbbc
I20260928 14:48:01.347239 22942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:48:01 dist-test-slave-j968 krb5kdc[22851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:01.352141 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.340322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.18.96.190:43455)
0928 14:48:01.340542 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.340551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.340577 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:48:01.341463 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.341470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.341542 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.341708 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:01.342536 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.343932 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.344996 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.345546 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.345672 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.348347 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:01.348351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:01.348351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:01.348363 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:01.350341 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.350783 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.350786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.350786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.350899 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.351128 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.351129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.351129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.351413 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:01.351566 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.351688 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.351817 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260928 14:48:01.355461 22942 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:01.356230 22942 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:01.362128 22942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910: Generated new TSK 0
I20260928 14:48:01.362346 22942 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:48:01.391741 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43455
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:01.476109 22957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:01.476359 22957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:01.476401 22957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:01.477619 22957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:01.477685 22957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:01.477732 22957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:01.479318 22957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:01.479599 22957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:01.479841 22957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:01.484979 22963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:01.485118 22957 server_base.cc:1034] running on GCE node
W20260928 14:48:01.484978 22962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:01.484977 22965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:01.485507 22957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:01.485798 22957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:01.487084 22957 hybrid_clock.cc:648] HybridClock initialized: now 1790606881487057 us; error 38 us; skew 500 ppm
Sep 28 14:48:01 dist-test-slave-j968 krb5kdc[22851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:01.489646 22957 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:48:01.490041 22957 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:01.490118 22957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:01.490267 22957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:01.491343 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "8ba5a26bd97340d98c2b268123d6a9ca"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "6c58dd4483f656de9fce9cf8feef0844"
server_key_iv: "c446d172782d7baf6bc73e49e619af20"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.491724 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "8ba5a26bd97340d98c2b268123d6a9ca"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "6c58dd4483f656de9fce9cf8feef0844"
server_key_iv: "c446d172782d7baf6bc73e49e619af20"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.493098 22957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:01.493860 22971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.494029 22957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:48:01.494127 22957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "8ba5a26bd97340d98c2b268123d6a9ca"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "6c58dd4483f656de9fce9cf8feef0844"
server_key_iv: "c446d172782d7baf6bc73e49e619af20"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.494210 22957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:01.503479 22957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:01.505450 22957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:01.505613 22957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:01.505897 22957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:01.506222 22957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:01.506275 22957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.506307 22957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:01.506336 22957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.512090 22957 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:45221
I20260928 14:48:01.512125 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:45221 every 8 connection(s)
I20260928 14:48:01.512424 22957 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:01.517284 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 22957
I20260928 14:48:01.517412 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:01.517673 18818 external_mini_cluster.cc:1428] Setting key 4672f76ea9dc7cf4b5e4b6d2d4c5226e
Sep 28 14:48:01 dist-test-slave-j968 krb5kdc[22851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:01.523669 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.513160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43455 (local address 127.18.96.129:47391)
0928 14:48:01.513309 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:01.513343 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:01.513845 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:01.514050 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:01.514053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:01.514316 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:01.514828 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.514833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.515568 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.515568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.516967 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.516970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.517566 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.517567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.517723 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.518331 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:01.518340 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:01.520165 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:01.522068 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:01.522074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:01.522076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:01.522303 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:01.522527 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:01.522528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:01.522528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:01.522675 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:01.523131 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:01.523138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:01.523266 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 14:48:01.523360 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 14:48:01.523498 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260928 14:48:01.523659 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.513177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:47391 (local address 127.18.96.190:43455)
0928 14:48:01.513227 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.513229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.513431 (+ 202us) server_negotiation.cc:407] Connection header received
0928 14:48:01.513930 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.513931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.513941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.513992 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:01.514937 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.515452 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.517080 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.517410 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.517484 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.520270 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:01.520272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:01.520272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:01.520286 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:01.521968 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.522379 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.522380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.522381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.522447 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.522744 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.522745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.522746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.522902 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:01.522979 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.523417 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.523521 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:48:01.524319 23085 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43455
I20260928 14:48:01.524422 23085 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:01.524596 23085 heartbeater.cc:511] Master 127.18.96.190:43455 requested a full tablet report, sending...
I20260928 14:48:01.525120 22895 ts_manager.cc:194] Registered new tserver with Master: 8ba5a26bd97340d98c2b268123d6a9ca (127.18.96.129:45221)
I20260928 14:48:01.525741 22895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:47391
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:48:01.546157 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43455
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:01.630432 23090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:01.630692 23090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:01.630733 23090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:01.632047 23090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:01.632112 23090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:01.632157 23090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:01.633754 23090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:01.634063 23090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:01.634315 23090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:01.639782 23096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:01.639782 23095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:01.639788 23098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:01.640252 23090 server_base.cc:1034] running on GCE node
I20260928 14:48:01.640537 23090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:01.640841 23090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:01.642001 23090 hybrid_clock.cc:648] HybridClock initialized: now 1790606881641971 us; error 51 us; skew 500 ppm
Sep 28 14:48:01 dist-test-slave-j968 krb5kdc[22851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:01.644793 23090 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:48:01.645165 23090 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:01.645215 23090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:01.645352 23090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:01.646576 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "e501f28d991b4e5f86bea62d9a0325b0"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "723508a24474cb6c58b6ae25222f6cee"
server_key_iv: "ce158b9de49d006c140050dc247caec7"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.647042 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "e501f28d991b4e5f86bea62d9a0325b0"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "723508a24474cb6c58b6ae25222f6cee"
server_key_iv: "ce158b9de49d006c140050dc247caec7"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.648562 23090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:01.649278 23104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.649426 23090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:01.649508 23090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "e501f28d991b4e5f86bea62d9a0325b0"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "723508a24474cb6c58b6ae25222f6cee"
server_key_iv: "ce158b9de49d006c140050dc247caec7"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.649564 23090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:01.666070 23090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:01.668092 23090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:01.668207 23090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:01.668388 23090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:01.668695 23090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:01.668728 23090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.668751 23090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:01.668795 23090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.674548 23090 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:35331
I20260928 14:48:01.674620 23217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:35331 every 8 connection(s)
I20260928 14:48:01.674902 23090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:01.681444 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 23090
I20260928 14:48:01.681551 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:01.681741 18818 external_mini_cluster.cc:1428] Setting key 581f22886e5ee146729c840f080546c4
Sep 28 14:48:01 dist-test-slave-j968 krb5kdc[22851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:01.685796 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.675746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:48329 (local address 127.18.96.190:43455)
0928 14:48:01.675821 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.675823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.676166 (+ 343us) server_negotiation.cc:407] Connection header received
0928 14:48:01.676682 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.676683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.676693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.676754 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:01.677787 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.678339 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.679563 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.679755 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.679813 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.682595 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:01.682596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:01.682597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:01.682606 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:01.684296 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.684841 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.684843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.684843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.684893 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.685180 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.685180 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.685181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.685281 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:01.685332 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.685670 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.685733 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:48:01.686117 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.675718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43455 (local address 127.18.96.130:48329)
0928 14:48:01.676040 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:01.676080 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:01.676590 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:01.676815 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:01.676818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:01.677137 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:01.677686 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.677691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.678457 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.678458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.679475 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.679477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.679840 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.679841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.680051 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.680701 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:01.680710 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:01.682492 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:01.684381 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:01.684389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:01.684391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:01.684759 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:01.684959 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:01.684960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:01.684961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:01.685112 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:01.685347 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:01.685353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:01.685526 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 14:48:01.685653 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 14:48:01.685833 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
I20260928 14:48:01.686739 23218 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43455
I20260928 14:48:01.686832 23218 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:01.687000 23218 heartbeater.cc:511] Master 127.18.96.190:43455 requested a full tablet report, sending...
I20260928 14:48:01.687405 22895 ts_manager.cc:194] Registered new tserver with Master: e501f28d991b4e5f86bea62d9a0325b0 (127.18.96.130:35331)
I20260928 14:48:01.687788 22895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:48329
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:48:01.707063 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43455
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:01.789330 23223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:01.789577 23223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:01.789614 23223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:01.790750 23223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:01.790829 23223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:01.790874 23223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:01.792397 23223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:01.792665 23223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:01.792892 23223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:01.798350 23229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:01.798341 23228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:01.798450 23223 server_base.cc:1034] running on GCE node
W20260928 14:48:01.798341 23231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:01.798794 23223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:01.799073 23223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:01.800215 23223 hybrid_clock.cc:648] HybridClock initialized: now 1790606881800199 us; error 33 us; skew 500 ppm
Sep 28 14:48:01 dist-test-slave-j968 krb5kdc[22851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:01.802850 23223 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:48:01.803236 23223 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:01.803287 23223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:01.803385 23223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:01.804510 23223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "6277182f4cd14fb79f4a9523814406e5"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "13cff4cf882d7041807f43199358c2e0"
server_key_iv: "dcc535f21862ee225525f23f8c93e106"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.804807 23223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "6277182f4cd14fb79f4a9523814406e5"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "13cff4cf882d7041807f43199358c2e0"
server_key_iv: "dcc535f21862ee225525f23f8c93e106"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.806034 23223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:48:01.806706 23237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.806891 23223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:01.806968 23223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "6277182f4cd14fb79f4a9523814406e5"
format_stamp: "Formatted at 2026-09-28 14:48:01 on dist-test-slave-j968"
server_key: "13cff4cf882d7041807f43199358c2e0"
server_key_iv: "dcc535f21862ee225525f23f8c93e106"
server_key_version: "encryptionkey@0"
I20260928 14:48:01.807061 23223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:01.814726 23223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:01.817065 23223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:01.817212 23223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:01.817438 23223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:01.817761 23223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:01.817821 23223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.817883 23223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:01.817911 23223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:01.823801 23223 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:43733
I20260928 14:48:01.823834 23350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:43733 every 8 connection(s)
I20260928 14:48:01.824201 23223 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:48:01 dist-test-slave-j968 krb5kdc[22851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606881, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:01.831940 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 23223
I20260928 14:48:01.832069 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:01.832331 18818 external_mini_cluster.cc:1428] Setting key 39e5dee5a2075a6baa556933b972e8ca
I20260928 14:48:01.834285 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.825019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:53605 (local address 127.18.96.190:43455)
0928 14:48:01.825096 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.825099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.825275 (+ 176us) server_negotiation.cc:407] Connection header received
0928 14:48:01.825826 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.825828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.825840 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.825891 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:01.826877 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.827446 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.828548 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.828758 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.828832 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.831062 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:01.831063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:01.831064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:01.831073 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:01.832991 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.833474 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.833475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.833476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.833530 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.833752 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.833753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.833753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.833866 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:01.833921 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.834149 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.834215 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:48:01.834484 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.824920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43455 (local address 127.18.96.131:53605)
0928 14:48:01.825139 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:01.825188 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:01.825703 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:01.825959 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:01.825963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:01.826233 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:01.826757 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.826763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.827544 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.827545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.828451 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.828453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.828848 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.828849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.828977 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.829443 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:01.829451 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:01.830887 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:01.833086 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:01.833092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:01.833094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:01.833393 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:01.833592 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:01.833592 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:01.833593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:01.833693 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:01.833926 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:01.833930 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:01.834042 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 14:48:01.834123 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:48:01.834241 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":115,"threads_started":1}
I20260928 14:48:01.834991 23351 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43455
I20260928 14:48:01.835086 23351 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:01.835268 23351 heartbeater.cc:511] Master 127.18.96.190:43455 requested a full tablet report, sending...
I20260928 14:48:01.835680 22895 ts_manager.cc:194] Registered new tserver with Master: 6277182f4cd14fb79f4a9523814406e5 (127.18.96.131:43733)
I20260928 14:48:01.836134 22895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:53605
I20260928 14:48:01.844430 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:48:01.852027 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.845820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.18.96.190:43455)
0928 14:48:01.845876 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.845878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.846052 (+ 174us) server_negotiation.cc:407] Connection header received
0928 14:48:01.846132 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.846133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.846143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.846188 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:01.846768 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.847262 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.848097 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.848293 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.848375 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.849606 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:01.849608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:01.849608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:01.849616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:01.850819 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.851177 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.851179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.851179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.851240 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:01.851435 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:01.851436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:01.851436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:01.851601 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:01.851710 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.851812 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.851927 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:48:01.853665 22895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:01.855193 22895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:01.861819 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.857923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.18.96.130:35331)
0928 14:48:01.858336 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.858341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.858352 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:01.858392 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.858394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.858409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.858481 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:01.858681 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.859372 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.860955 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.861481 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.861548 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.861605 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.861701 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.861741 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":82,"threads_started":1}
I20260928 14:48:01.861922 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.857495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:45221 (local address 127.0.0.1:38452)
0928 14:48:01.857749 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:01.857760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:01.857812 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:01.858255 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:01.858255 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:01.858257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:01.858315 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.858316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.859525 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.859526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.860822 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.860825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.861707 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.861708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.861759 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.861762 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:01.861790 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 14:48:01.861821 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":83,"thread_start_us":46,"threads_started":1}
I20260928 14:48:01.861831 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.857675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:35331 (local address 127.0.0.1:53202)
0928 14:48:01.858087 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:01.858098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:01.858143 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:01.858575 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:01.858576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:01.858577 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:01.858607 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.858607 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.859494 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.859495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.860860 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.860862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.861560 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.861561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.861623 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.861628 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:48:01.861688 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 14:48:01.861734 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":69,"thread_start_us":42,"threads_started":1}
I20260928 14:48:01.862098 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.857840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.18.96.129:45221)
0928 14:48:01.858074 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.858080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.858090 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:01.858127 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.858129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.858145 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.858253 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:01.858378 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.859413 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.860911 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.861615 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.861885 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.861954 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.861982 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.862036 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":142,"threads_started":1}
I20260928 14:48:01.862272 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.857431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:43733 (local address 127.0.0.1:59058)
0928 14:48:01.857570 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:01.857611 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:01.857673 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:01.857968 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:01.857970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:01.857972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:01.858103 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.858105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.859447 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.859448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.860594 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.860596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.861460 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.861462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.861556 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.861572 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 14:48:01.861640 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:48:01.861701 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
I20260928 14:48:01.862406 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.857562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.18.96.131:43733)
0928 14:48:01.857708 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.857714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.857725 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:01.857768 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.857771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.857788 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.857867 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:01.858195 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.859334 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.860677 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.861357 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.861435 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.861492 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.861778 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.861821 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 14:48:01.862901 23019 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8ddc7d616e4ba0be18a117a6bd161c (DEFAULT_TABLE table=test-table [id=4223e58b471d42d394adabba2a34ae89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:01.862920 23285 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8ddc7d616e4ba0be18a117a6bd161c (DEFAULT_TABLE table=test-table [id=4223e58b471d42d394adabba2a34ae89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:01.863270 23285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8ddc7d616e4ba0be18a117a6bd161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:01.863270 23019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8ddc7d616e4ba0be18a117a6bd161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:01.866621 23366 tablet_bootstrap.cc:492] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca: Bootstrap starting.
I20260928 14:48:01.867221 23366 tablet_bootstrap.cc:654] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:01.867151 23152 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8ddc7d616e4ba0be18a117a6bd161c (DEFAULT_TABLE table=test-table [id=4223e58b471d42d394adabba2a34ae89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:01.867473 23152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8ddc7d616e4ba0be18a117a6bd161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:01.867754 23366 log.cc:824] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:01.868039 23365 tablet_bootstrap.cc:492] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: Bootstrap starting.
I20260928 14:48:01.868610 23366 tablet_bootstrap.cc:492] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca: No bootstrap required, opened a new log
I20260928 14:48:01.868690 23365 tablet_bootstrap.cc:654] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:01.868688 23366 ts_tablet_manager.cc:1397] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:48:01.869256 23365 log.cc:824] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:01.870000 23365 tablet_bootstrap.cc:492] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: No bootstrap required, opened a new log
I20260928 14:48:01.870065 23365 ts_tablet_manager.cc:1397] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:01.870450 23369 tablet_bootstrap.cc:492] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: Bootstrap starting.
I20260928 14:48:01.870890 23366 raft_consensus.cc:348] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } }
I20260928 14:48:01.871062 23369 tablet_bootstrap.cc:654] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:01.871089 23366 raft_consensus.cc:374] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:01.871129 23366 raft_consensus.cc:729] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba5a26bd97340d98c2b268123d6a9ca, State: Initialized, Role: FOLLOWER
I20260928 14:48:01.871258 23366 consensus_queue.cc:260] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } }
I20260928 14:48:01.871524 23366 ts_tablet_manager.cc:1428] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:01.871671 23085 heartbeater.cc:503] Master 127.18.96.190:43455 was elected leader, sending a full tablet report...
I20260928 14:48:01.871673 23369 log.cc:824] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:01.872139 23365 raft_consensus.cc:348] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } }
I20260928 14:48:01.872308 23365 raft_consensus.cc:374] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:01.872351 23365 raft_consensus.cc:729] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6277182f4cd14fb79f4a9523814406e5, State: Initialized, Role: FOLLOWER
I20260928 14:48:01.872493 23369 tablet_bootstrap.cc:492] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: No bootstrap required, opened a new log
I20260928 14:48:01.872468 23365 consensus_queue.cc:260] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } }
I20260928 14:48:01.872553 23369 ts_tablet_manager.cc:1397] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:48:01.872730 23351 heartbeater.cc:503] Master 127.18.96.190:43455 was elected leader, sending a full tablet report...
I20260928 14:48:01.872728 23365 ts_tablet_manager.cc:1428] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:48:01.874287 23369 raft_consensus.cc:348] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } }
I20260928 14:48:01.874410 23369 raft_consensus.cc:374] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:01.874432 23369 raft_consensus.cc:729] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e501f28d991b4e5f86bea62d9a0325b0, State: Initialized, Role: FOLLOWER
I20260928 14:48:01.874534 23369 consensus_queue.cc:260] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } }
I20260928 14:48:01.874758 23369 ts_tablet_manager.cc:1428] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 14:48:01.874811 23218 heartbeater.cc:503] Master 127.18.96.190:43455 was elected leader, sending a full tablet report...
I20260928 14:48:01.880539 23370 raft_consensus.cc:482] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:01.880667 23370 raft_consensus.cc:504] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } }
I20260928 14:48:01.881044 23370 leader_election.cc:288] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6277182f4cd14fb79f4a9523814406e5 (127.18.96.131:43733), e501f28d991b4e5f86bea62d9a0325b0 (127.18.96.130:35331)
I20260928 14:48:01.883988 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.881255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:60335 (local address 127.18.96.131:43733)
0928 14:48:01.881299 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.881301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.881522 (+ 221us) server_negotiation.cc:407] Connection header received
0928 14:48:01.881604 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.881606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.881620 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.881667 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:01.881829 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.882288 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.883252 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.883615 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.883693 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.883703 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.883879 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.883919 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:48:01.883996 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.881345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:59807 (local address 127.18.96.130:35331)
0928 14:48:01.881403 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.881406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.881416 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:01.881449 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.881450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.881466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.881522 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:01.881729 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.882272 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.883260 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.883615 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.883694 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.883703 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.883896 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.883929 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:48:01.884012 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.881219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:35331 (local address 127.18.96.129:59807)
0928 14:48:01.881307 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:01.881325 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:01.881371 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:01.881565 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:01.881566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:01.881568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:01.881650 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.881651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.882362 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.882363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.883158 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.883160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.883701 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.883702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.883787 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.883790 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:01.883880 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 14:48:01.883923 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":31}
I20260928 14:48:01.884022 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.881219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:43733 (local address 127.18.96.129:60335)
0928 14:48:01.881431 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:01.881442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:01.881547 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:01.881668 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:01.881669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:01.881671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:01.881754 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.881756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.882384 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.882385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.883162 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:01.883164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.883702 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.883703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:01.883777 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.883780 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:01.883852 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:48:01.883919 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":22,"thread_start_us":112,"threads_started":1}
I20260928 14:48:01.884331 23305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8ddc7d616e4ba0be18a117a6bd161c" candidate_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6277182f4cd14fb79f4a9523814406e5" is_pre_election: true
I20260928 14:48:01.884439 23305 raft_consensus.cc:2454] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba5a26bd97340d98c2b268123d6a9ca in term 0.
I20260928 14:48:01.884645 22973 leader_election.cc:302] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6277182f4cd14fb79f4a9523814406e5, 8ba5a26bd97340d98c2b268123d6a9ca; no voters:
I20260928 14:48:01.884656 23172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8ddc7d616e4ba0be18a117a6bd161c" candidate_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e501f28d991b4e5f86bea62d9a0325b0" is_pre_election: true
I20260928 14:48:01.884776 23172 raft_consensus.cc:2454] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba5a26bd97340d98c2b268123d6a9ca in term 0.
I20260928 14:48:01.884773 23370 raft_consensus.cc:2781] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:01.884814 23370 raft_consensus.cc:482] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:01.884835 23370 raft_consensus.cc:3037] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:01.885531 23370 raft_consensus.cc:504] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } }
I20260928 14:48:01.885656 23370 leader_election.cc:288] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [CANDIDATE]: Term 1 election: Requested vote from peers 6277182f4cd14fb79f4a9523814406e5 (127.18.96.131:43733), e501f28d991b4e5f86bea62d9a0325b0 (127.18.96.130:35331)
I20260928 14:48:01.885773 23305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8ddc7d616e4ba0be18a117a6bd161c" candidate_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6277182f4cd14fb79f4a9523814406e5"
I20260928 14:48:01.885795 23172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8ddc7d616e4ba0be18a117a6bd161c" candidate_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e501f28d991b4e5f86bea62d9a0325b0"
I20260928 14:48:01.885851 23305 raft_consensus.cc:3037] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:01.885864 23172 raft_consensus.cc:3037] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:01.886533 23172 raft_consensus.cc:2454] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba5a26bd97340d98c2b268123d6a9ca in term 1.
I20260928 14:48:01.886675 23305 raft_consensus.cc:2454] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba5a26bd97340d98c2b268123d6a9ca in term 1.
I20260928 14:48:01.886694 22974 leader_election.cc:302] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba5a26bd97340d98c2b268123d6a9ca, e501f28d991b4e5f86bea62d9a0325b0; no voters:
I20260928 14:48:01.886802 23370 raft_consensus.cc:2781] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:01.887076 23370 raft_consensus.cc:686] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [term 1 LEADER]: Becoming Leader. State: Replica: 8ba5a26bd97340d98c2b268123d6a9ca, State: Running, Role: LEADER
I20260928 14:48:01.887216 23370 consensus_queue.cc:237] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } }
I20260928 14:48:01.888058 22893 catalog_manager.cc:5495] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba5a26bd97340d98c2b268123d6a9ca (127.18.96.129). New cstate: current_term: 1 leader_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba5a26bd97340d98c2b268123d6a9ca" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 45221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 } health_report { overall_health: UNKNOWN } } }
I20260928 14:48:01.896687 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.894427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.18.96.129:45221)
0928 14:48:01.894529 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:01.894530 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:01.894538 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:48:01.894566 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:01.894567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:01.894581 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:01.894645 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:01.894743 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.895176 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.895856 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:01.896117 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:01.896179 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:01.896305 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:01.896394 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:01.896476 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:01.896528 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 14:48:01.896560 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:48:01.898712 23305 raft_consensus.cc:1261] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [term 1 FOLLOWER]: Refusing update from remote peer 8ba5a26bd97340d98c2b268123d6a9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:01.898711 23172 raft_consensus.cc:1261] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [term 1 FOLLOWER]: Refusing update from remote peer 8ba5a26bd97340d98c2b268123d6a9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:01.899030 23370 consensus_queue.cc:1035] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e501f28d991b4e5f86bea62d9a0325b0" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:01.899151 23375 consensus_queue.cc:1035] T 8a8ddc7d616e4ba0be18a117a6bd161c P 8ba5a26bd97340d98c2b268123d6a9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "6277182f4cd14fb79f4a9523814406e5" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:01.901196 23378 mvcc.cc:205] Tried to move back new op lower bound from 7334325788254265344 to 7334325788210606080. Current Snapshot: MvccSnapshot[applied={T|T < 7334325788254265344}]
I20260928 14:48:01.901563 23379 mvcc.cc:205] Tried to move back new op lower bound from 7334325788254265344 to 7334325788210606080. Current Snapshot: MvccSnapshot[applied={T|T < 7334325788254265344}]
I20260928 14:48:01.902501 23380 mvcc.cc:205] Tried to move back new op lower bound from 7334325788254265344 to 7334325788210606080. Current Snapshot: MvccSnapshot[applied={T|T < 7334325788254265344}]
I20260928 14:48:01.903764 22894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:01.903903 22894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:01.904767 22894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80183b8859dc47948eefd23ae9bc6910: Sending DeleteTablet for 3 replicas of tablet 8a8ddc7d616e4ba0be18a117a6bd161c
I20260928 14:48:01.905148 23285 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8ddc7d616e4ba0be18a117a6bd161c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:01 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:59058
I20260928 14:48:01.905259 23152 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8ddc7d616e4ba0be18a117a6bd161c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:01 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:53202
I20260928 14:48:01.905444 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 22957
I20260928 14:48:01.905509 23019 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8ddc7d616e4ba0be18a117a6bd161c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:01 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:38452
I20260928 14:48:01.908033 23389 tablet_replica.cc:330] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: stopping tablet replica
I20260928 14:48:01.908164 23389 raft_consensus.cc:2229] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:01.908308 23389 raft_consensus.cc:2258] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:01.908666 23388 tablet_replica.cc:330] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: stopping tablet replica
I20260928 14:48:01.908774 23388 raft_consensus.cc:2229] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:01.908783 23389 ts_tablet_manager.cc:1905] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:01.908876 23388 raft_consensus.cc:2258] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:01.909301 23388 ts_tablet_manager.cc:1905] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:01.910533 23389 ts_tablet_manager.cc:1918] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:01.910610 23389 log.cc:1196] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/8a8ddc7d616e4ba0be18a117a6bd161c
I20260928 14:48:01.910897 23389 ts_tablet_manager.cc:1939] T 8a8ddc7d616e4ba0be18a117a6bd161c P e501f28d991b4e5f86bea62d9a0325b0: Deleting consensus metadata
I20260928 14:48:01.910974 23388 ts_tablet_manager.cc:1918] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:01.911044 23388 log.cc:1196] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/8a8ddc7d616e4ba0be18a117a6bd161c
I20260928 14:48:01.911288 23388 ts_tablet_manager.cc:1939] T 8a8ddc7d616e4ba0be18a117a6bd161c P 6277182f4cd14fb79f4a9523814406e5: Deleting consensus metadata
I20260928 14:48:01.911831 22881 catalog_manager.cc:4841] TS e501f28d991b4e5f86bea62d9a0325b0 (127.18.96.130:35331): tablet 8a8ddc7d616e4ba0be18a117a6bd161c (table test-table [id=4223e58b471d42d394adabba2a34ae89]) successfully deleted
W20260928 14:48:01.912107 22881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:45221 (error 108)
I20260928 14:48:01.912397 22880 catalog_manager.cc:4841] TS 6277182f4cd14fb79f4a9523814406e5 (127.18.96.131:43733): tablet 8a8ddc7d616e4ba0be18a117a6bd161c (table test-table [id=4223e58b471d42d394adabba2a34ae89]) successfully deleted
I20260928 14:48:01.912614 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:01.912500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:45221 (local address 127.0.0.1:38472)
0928 14:48:01.912541 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:01.912581 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:45221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 14:48:01.912719 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 23090
W20260928 14:48:01.912739 22881 catalog_manager.cc:4584] TS 8ba5a26bd97340d98c2b268123d6a9ca (127.18.96.129:45221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a8ddc7d616e4ba0be18a117a6bd161c: Network error: Client connection negotiation failed: client connection to 127.18.96.129:45221: connect: Connection refused (error 111)
I20260928 14:48:01.918679 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 23223
I20260928 14:48:01.923306 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 22864
2026-09-28T14:48:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1886 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T14:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:01Z Disabled control of system clock
I20260928 14:48:01.952349 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34991
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:34991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:48:02.034864 23394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:02.035108 23394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:02.035132 23394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:02.036584 23394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:02.036638 23394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:02.036651 23394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:02.036664 23394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:02.036676 23394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:02.038236 23394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:34991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:02.038579 23394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:02.038924 23394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:02.044193 23400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:02.044211 23399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.044281 23394 server_base.cc:1034] running on GCE node
W20260928 14:48:02.044212 23402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.044780 23394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:02.045133 23394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:02.046320 23394 hybrid_clock.cc:648] HybridClock initialized: now 1790606882046277 us; error 62 us; skew 500 ppm
I20260928 14:48:02.047824 23394 webserver.cc:460] Webserver started at http://127.18.96.190:44399/ using document root <none> and password file <none>
I20260928 14:48:02.048205 23394 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:02.048282 23394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:02.048440 23394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:02.049566 23394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "fb998d2547b84720b2531eeeb6e3538c"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "86d7b750b88a48244b323323101421da"
server_key_iv: "63a4ce62848520bfd54185efcefebea5"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.049933 23394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "fb998d2547b84720b2531eeeb6e3538c"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "86d7b750b88a48244b323323101421da"
server_key_iv: "63a4ce62848520bfd54185efcefebea5"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.051301 23394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:02.052196 23408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.052378 23394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:02.052490 23394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "fb998d2547b84720b2531eeeb6e3538c"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "86d7b750b88a48244b323323101421da"
server_key_iv: "63a4ce62848520bfd54185efcefebea5"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.052579 23394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:02.056087 23394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:02.056348 23394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:02.056499 23394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:02.060446 23394 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:34991
I20260928 14:48:02.060478 23460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:34991 every 8 connection(s)
I20260928 14:48:02.060840 23394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:02.061517 23461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:02.064725 23461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c: Bootstrap starting.
I20260928 14:48:02.065410 23461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:02.065665 23461 log.cc:824] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:02.066363 23461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c: No bootstrap required, opened a new log
I20260928 14:48:02.066972 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 23394
I20260928 14:48:02.067128 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:02.067329 18818 external_mini_cluster.cc:1428] Setting key acfd9d7a92a0620e611819093a3e0bf0
I20260928 14:48:02.068552 23461 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb998d2547b84720b2531eeeb6e3538c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34991 } }
I20260928 14:48:02.068760 23461 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:02.068818 23461 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb998d2547b84720b2531eeeb6e3538c, State: Initialized, Role: FOLLOWER
I20260928 14:48:02.068966 23461 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb998d2547b84720b2531eeeb6e3538c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34991 } }
I20260928 14:48:02.069087 23461 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:02.069129 23461 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:02.069198 23461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:02.070051 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.067960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38400 (local address 127.18.96.190:34991)
0928 14:48:02.068236 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.068244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.068269 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:48:02.068506 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.068513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.068555 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.068668 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.069566 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.069570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.069570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.069574 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.069613 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.069624 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.069680 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.069774 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.069840 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":142,"threads_started":1}
I20260928 14:48:02.070039 23461 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb998d2547b84720b2531eeeb6e3538c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34991 } }
I20260928 14:48:02.070199 23461 leader_election.cc:302] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb998d2547b84720b2531eeeb6e3538c; no voters:
I20260928 14:48:02.070379 23461 leader_election.cc:288] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:02.070657 23461 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:02.070691 23466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:02.070806 23466 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [term 1 LEADER]: Becoming Leader. State: Replica: fb998d2547b84720b2531eeeb6e3538c, State: Running, Role: LEADER
I20260928 14:48:02.071038 23466 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb998d2547b84720b2531eeeb6e3538c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34991 } }
I20260928 14:48:02.071414 23466 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb998d2547b84720b2531eeeb6e3538c. Latest consensus state: current_term: 1 leader_uuid: "fb998d2547b84720b2531eeeb6e3538c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb998d2547b84720b2531eeeb6e3538c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34991 } } }
I20260928 14:48:02.071491 23466 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:02.071805 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb998d2547b84720b2531eeeb6e3538c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb998d2547b84720b2531eeeb6e3538c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34991 } } }
I20260928 14:48:02.071903 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:02.072192 23475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:02.072674 23475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:02.074281 23475 catalog_manager.cc:1347] Generated new cluster ID: 6d9af6f9b68a438a9e996a60706c55c6
I20260928 14:48:02.074339 23475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:02.082121 23475 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:02.082309 23475 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:02.085417 23475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c: Generated new TSK 0
I20260928 14:48:02.085611 23475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:48:02.087016 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34991
--builtin_ntp_servers=127.18.96.148:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:48:02.173523 23485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:02.173733 23485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:02.173756 23485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:02.174957 23485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:02.175000 23485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:02.175028 23485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:02.176455 23485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:02.176704 23485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:02.177016 23485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:02.182304 23490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:02.182292 23491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.182698 23485 server_base.cc:1034] running on GCE node
W20260928 14:48:02.182734 23493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.182972 23485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:02.183274 23485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:02.184434 23485 hybrid_clock.cc:648] HybridClock initialized: now 1790606882184419 us; error 32 us; skew 500 ppm
I20260928 14:48:02.185719 23485 webserver.cc:460] Webserver started at http://127.18.96.129:37833/ using document root <none> and password file <none>
I20260928 14:48:02.185958 23485 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:02.186084 23485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:02.186235 23485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:02.187309 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "7d77867f58c3452eb2433eb54586c15c"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "87dcf77d2673e453e5976a8c87c2cab6"
server_key_iv: "c17b5156e930fe08e2c85bd2ca2d0bda"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.187667 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "7d77867f58c3452eb2433eb54586c15c"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "87dcf77d2673e453e5976a8c87c2cab6"
server_key_iv: "c17b5156e930fe08e2c85bd2ca2d0bda"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.189124 23485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:48:02.189932 23499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.190192 23485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:02.190299 23485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "7d77867f58c3452eb2433eb54586c15c"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "87dcf77d2673e453e5976a8c87c2cab6"
server_key_iv: "c17b5156e930fe08e2c85bd2ca2d0bda"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.190379 23485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:02.195757 23485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:02.196166 23485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:02.196336 23485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:02.196595 23485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:02.197002 23485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:02.197067 23485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.197103 23485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:02.197131 23485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.204999 23485 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:34393
I20260928 14:48:02.205053 23612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:34393 every 8 connection(s)
I20260928 14:48:02.205472 23485 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:02.208931 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.206389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:59349 (local address 127.18.96.190:34991)
0928 14:48:02.206457 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.206460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.206735 (+ 275us) server_negotiation.cc:407] Connection header received
0928 14:48:02.207293 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.207294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.207302 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.207361 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.208661 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.208663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.208663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.208668 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.208686 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.208690 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.208744 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.208839 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.208857 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:48:02.209152 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.206374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34991 (local address 127.18.96.129:59349)
0928 14:48:02.206590 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.206633 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:02.207185 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:02.207443 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:02.207447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:02.207464 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:02.208493 (+ 1029us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:02.208494 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:02.208525 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:02.208556 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:02.208740 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:02.208746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:02.208749 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:02.208850 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:48:02.208916 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260928 14:48:02.209731 23613 heartbeater.cc:347] Connected to a master server at 127.18.96.190:34991
I20260928 14:48:02.209841 23613 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:02.210000 23613 heartbeater.cc:511] Master 127.18.96.190:34991 requested a full tablet report, sending...
I20260928 14:48:02.210541 23425 ts_manager.cc:194] Registered new tserver with Master: 7d77867f58c3452eb2433eb54586c15c (127.18.96.129:34393)
I20260928 14:48:02.212406 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 23485
I20260928 14:48:02.212533 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:02.212714 18818 external_mini_cluster.cc:1428] Setting key adf6dd570c59ce79cfbd40a6ade8e09c
I20260928 14:48:02.213958 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34991
--builtin_ntp_servers=127.18.96.148:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:48:02.297964 23616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:02.298203 23616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:02.298242 23616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:02.299474 23616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:02.299540 23616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:02.299584 23616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:02.300974 23616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:02.301231 23616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:02.301529 23616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:02.307840 23621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.308017 23616 server_base.cc:1034] running on GCE node
W20260928 14:48:02.307843 23624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:02.307839 23622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.308441 23616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:02.308748 23616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:02.309957 23616 hybrid_clock.cc:648] HybridClock initialized: now 1790606882309939 us; error 47 us; skew 500 ppm
I20260928 14:48:02.311348 23616 webserver.cc:460] Webserver started at http://127.18.96.130:43511/ using document root <none> and password file <none>
I20260928 14:48:02.311610 23616 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:02.311681 23616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:02.311838 23616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:02.313004 23616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "24a0db6193804fd286d2c86b456c9025"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "5d2a6e49293c5400e3ce68e89cb17d51"
server_key_iv: "637dc0ae35b04bd6314c518da38ee025"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.313406 23616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "24a0db6193804fd286d2c86b456c9025"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "5d2a6e49293c5400e3ce68e89cb17d51"
server_key_iv: "637dc0ae35b04bd6314c518da38ee025"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.314719 23616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:02.315416 23630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.315598 23616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:48:02.315703 23616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "24a0db6193804fd286d2c86b456c9025"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "5d2a6e49293c5400e3ce68e89cb17d51"
server_key_iv: "637dc0ae35b04bd6314c518da38ee025"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.315778 23616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:02.319346 23616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:02.319630 23616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:02.319779 23616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:02.320176 23616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:02.320689 23616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:02.320809 23616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.320895 23616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:02.320926 23616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.326968 23616 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:46029
I20260928 14:48:02.327010 23743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:46029 every 8 connection(s)
I20260928 14:48:02.327400 23616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:02.329259 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 23616
I20260928 14:48:02.329420 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:02.329635 18818 external_mini_cluster.cc:1428] Setting key 7700446303167e2ac9e442c2b69b577b
I20260928 14:48:02.330765 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34991
--builtin_ntp_servers=127.18.96.148:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 14:48:02.331121 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.328282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:35669 (local address 127.18.96.190:34991)
0928 14:48:02.328353 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.328355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.328559 (+ 204us) server_negotiation.cc:407] Connection header received
0928 14:48:02.329110 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.329112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.329121 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.329200 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.330797 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.330798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.330799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.330803 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.330839 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.330845 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.330912 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.331028 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.331045 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:48:02.331342 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.328270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34991 (local address 127.18.96.130:35669)
0928 14:48:02.328450 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.328485 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:02.328937 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:02.329326 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:02.329331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:02.329339 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:02.330614 (+ 1275us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:02.330615 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:02.330658 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:02.330697 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:02.330946 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:02.330953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:02.330958 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:48:02.331038 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:48:02.331106 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260928 14:48:02.331802 23744 heartbeater.cc:347] Connected to a master server at 127.18.96.190:34991
I20260928 14:48:02.331933 23744 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:02.332099 23744 heartbeater.cc:511] Master 127.18.96.190:34991 requested a full tablet report, sending...
I20260928 14:48:02.332557 23425 ts_manager.cc:194] Registered new tserver with Master: 24a0db6193804fd286d2c86b456c9025 (127.18.96.130:46029)
W20260928 14:48:02.412014 23747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:02.412228 23747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:02.412251 23747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:02.413471 23747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:02.413511 23747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:02.413537 23747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:02.414877 23747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:02.415138 23747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:02.415432 23747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:02.420724 23752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:02.420733 23755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.420763 23747 server_base.cc:1034] running on GCE node
W20260928 14:48:02.420722 23753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.421240 23747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:02.421509 23747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:02.422685 23747 hybrid_clock.cc:648] HybridClock initialized: now 1790606882422657 us; error 49 us; skew 500 ppm
I20260928 14:48:02.424044 23747 webserver.cc:460] Webserver started at http://127.18.96.131:37143/ using document root <none> and password file <none>
I20260928 14:48:02.424310 23747 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:02.424379 23747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:02.424520 23747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:02.425696 23747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "073424376b7246f9b4e18071e118e2b3"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "2743efb19369c0c06d2f4888dac1b79d"
server_key_iv: "e0d0c368557fb83f1acc504bdc9186ae"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.426069 23747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "073424376b7246f9b4e18071e118e2b3"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "2743efb19369c0c06d2f4888dac1b79d"
server_key_iv: "e0d0c368557fb83f1acc504bdc9186ae"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.427431 23747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:02.428315 23761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.428493 23747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:48:02.428611 23747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "073424376b7246f9b4e18071e118e2b3"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "2743efb19369c0c06d2f4888dac1b79d"
server_key_iv: "e0d0c368557fb83f1acc504bdc9186ae"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.428690 23747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:02.432314 23747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:02.432585 23747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:02.432723 23747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:02.432937 23747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:02.433246 23747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:02.433300 23747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.433339 23747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:02.433365 23747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.439323 23747 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:35547
I20260928 14:48:02.439397 23874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:35547 every 8 connection(s)
I20260928 14:48:02.439769 23747 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:02.443145 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.440658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:60635 (local address 127.18.96.190:34991)
0928 14:48:02.440715 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.440717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.440981 (+ 264us) server_negotiation.cc:407] Connection header received
0928 14:48:02.441523 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.441525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.441531 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.441599 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.442894 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.442896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.442896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.442900 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.442916 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.442921 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.442983 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.443069 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.443084 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:48:02.443356 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.440541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34991 (local address 127.18.96.131:60635)
0928 14:48:02.440769 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.440806 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:02.441367 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:02.441733 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:02.441737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:02.441745 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:02.442742 (+ 997us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:02.442742 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:02.442771 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:02.442798 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:02.442993 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:02.442997 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:02.443000 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:02.443061 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 14:48:02.443113 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260928 14:48:02.443897 23875 heartbeater.cc:347] Connected to a master server at 127.18.96.190:34991
I20260928 14:48:02.443992 23875 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:02.444111 23875 heartbeater.cc:511] Master 127.18.96.190:34991 requested a full tablet report, sending...
I20260928 14:48:02.444487 23425 ts_manager.cc:194] Registered new tserver with Master: 073424376b7246f9b4e18071e118e2b3 (127.18.96.131:35547)
I20260928 14:48:02.446081 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 23747
I20260928 14:48:02.446194 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:02.446372 18818 external_mini_cluster.cc:1428] Setting key 0d69c59bb943eaea470562a2f0eb9db7
I20260928 14:48:02.447280 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:48:02.449870 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.448657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.18.96.190:34991)
0928 14:48:02.448722 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.448724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.448812 (+ 88us) server_negotiation.cc:407] Connection header received
0928 14:48:02.448894 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.448895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.448901 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.448961 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.449650 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.449651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.449652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.449656 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.449672 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.449676 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.449725 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.449780 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.449796 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:48:02.451531 23425 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:02.453173 23425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:02.457266 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.455399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:35547 (local address 127.0.0.1:47522)
0928 14:48:02.455566 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.455606 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:02.455661 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:02.456235 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:02.456241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:02.456248 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:02.456771 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:02.456771 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:02.456788 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:02.456814 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:02.457097 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:02.457104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:02.457106 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:48:02.457173 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:48:02.457187 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260928 14:48:02.457327 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.455601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.18.96.131:35547)
0928 14:48:02.455785 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.455788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.455798 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:02.455825 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.455827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.455837 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.455926 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.456914 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.456923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.456924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.456928 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.456969 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.456997 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.457069 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.457205 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.457225 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260928 14:48:02.457368 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.455642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:34393 (local address 127.0.0.1:44290)
0928 14:48:02.455919 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.455930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:02.456033 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:02.456309 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:02.456310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:02.456312 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:02.456947 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:02.456948 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:02.456955 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:02.456967 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:02.457241 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:02.457242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:02.457242 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 14:48:02.457286 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:48:02.457298 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":74,"threads_started":1}
I20260928 14:48:02.457542 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.455857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.18.96.129:34393)
0928 14:48:02.456178 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.456182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.456192 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:02.456218 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.456220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.456229 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.456295 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.457054 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.457064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.457064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.457069 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.457101 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.457127 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.457213 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.457442 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.457469 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":88,"threads_started":1}
I20260928 14:48:02.457962 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.455406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:46029 (local address 127.0.0.1:43526)
0928 14:48:02.455757 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.455768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:02.455850 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:02.456406 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:02.456407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:02.456410 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:02.457599 (+ 1189us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:02.457600 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:02.457606 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:02.457617 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:02.457832 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:02.457833 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:02.457834 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:48:02.457882 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 14:48:02.457895 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":82,"threads_started":1}
I20260928 14:48:02.457983 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.455512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.18.96.130:46029)
0928 14:48:02.455703 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.455709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.455828 (+ 119us) server_negotiation.cc:407] Connection header received
0928 14:48:02.456043 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.456050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.456061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.456114 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.457687 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.457696 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.457697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.457702 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.457735 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.457759 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.457820 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.457887 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.457903 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260928 14:48:02.458033 23809 tablet_service.cc:1467] Processing CreateTablet for tablet 79d8c36e229247ac90c080521793ea4d (DEFAULT_TABLE table=test-table [id=96e53fb2e5474768b1f052788552abf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:02.458242 23547 tablet_service.cc:1467] Processing CreateTablet for tablet 79d8c36e229247ac90c080521793ea4d (DEFAULT_TABLE table=test-table [id=96e53fb2e5474768b1f052788552abf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:02.458396 23809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d8c36e229247ac90c080521793ea4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:02.458527 23547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d8c36e229247ac90c080521793ea4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:02.461664 23890 tablet_bootstrap.cc:492] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: Bootstrap starting.
I20260928 14:48:02.461658 23889 tablet_bootstrap.cc:492] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c: Bootstrap starting.
I20260928 14:48:02.462383 23890 tablet_bootstrap.cc:654] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:02.462332 23678 tablet_service.cc:1467] Processing CreateTablet for tablet 79d8c36e229247ac90c080521793ea4d (DEFAULT_TABLE table=test-table [id=96e53fb2e5474768b1f052788552abf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:02.462448 23889 tablet_bootstrap.cc:654] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:02.462667 23678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d8c36e229247ac90c080521793ea4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:02.462750 23890 log.cc:824] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:02.462775 23889 log.cc:824] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:02.463512 23889 tablet_bootstrap.cc:492] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c: No bootstrap required, opened a new log
I20260928 14:48:02.463583 23889 ts_tablet_manager.cc:1397] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:48:02.463672 23890 tablet_bootstrap.cc:492] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: No bootstrap required, opened a new log
I20260928 14:48:02.463729 23890 ts_tablet_manager.cc:1397] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:48:02.465529 23893 tablet_bootstrap.cc:492] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: Bootstrap starting.
I20260928 14:48:02.466099 23893 tablet_bootstrap.cc:654] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:02.466241 23889 raft_consensus.cc:348] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } }
I20260928 14:48:02.466306 23890 raft_consensus.cc:348] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } }
I20260928 14:48:02.466423 23890 raft_consensus.cc:374] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:02.466389 23893 log.cc:824] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:02.466413 23889 raft_consensus.cc:374] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:02.466465 23889 raft_consensus.cc:729] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d77867f58c3452eb2433eb54586c15c, State: Initialized, Role: FOLLOWER
I20260928 14:48:02.466465 23890 raft_consensus.cc:729] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 073424376b7246f9b4e18071e118e2b3, State: Initialized, Role: FOLLOWER
I20260928 14:48:02.466599 23890 consensus_queue.cc:260] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } }
I20260928 14:48:02.466599 23889 consensus_queue.cc:260] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } }
I20260928 14:48:02.466857 23890 ts_tablet_manager.cc:1428] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:02.466857 23889 ts_tablet_manager.cc:1428] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 14:48:02.466985 23875 heartbeater.cc:503] Master 127.18.96.190:34991 was elected leader, sending a full tablet report...
I20260928 14:48:02.467273 23613 heartbeater.cc:503] Master 127.18.96.190:34991 was elected leader, sending a full tablet report...
I20260928 14:48:02.467970 23893 tablet_bootstrap.cc:492] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: No bootstrap required, opened a new log
I20260928 14:48:02.468043 23893 ts_tablet_manager.cc:1397] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:48:02.469820 23893 raft_consensus.cc:348] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } }
I20260928 14:48:02.469966 23893 raft_consensus.cc:374] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:02.470012 23893 raft_consensus.cc:729] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a0db6193804fd286d2c86b456c9025, State: Initialized, Role: FOLLOWER
I20260928 14:48:02.470157 23893 consensus_queue.cc:260] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } }
I20260928 14:48:02.470407 23893 ts_tablet_manager.cc:1428] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:02.470444 23744 heartbeater.cc:503] Master 127.18.96.190:34991 was elected leader, sending a full tablet report...
W20260928 14:48:02.577981 23745 tablet.cc:2369] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:02.675411 23896 raft_consensus.cc:482] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:02.675608 23896 raft_consensus.cc:504] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } }
I20260928 14:48:02.676128 23896 leader_election.cc:288] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 073424376b7246f9b4e18071e118e2b3 (127.18.96.131:35547), 24a0db6193804fd286d2c86b456c9025 (127.18.96.130:46029)
I20260928 14:48:02.677600 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.676424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:52533 (local address 127.18.96.131:35547)
0928 14:48:02.676504 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.676507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.676518 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:02.676550 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.676551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.676558 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.676661 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.677361 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.677363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.677363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.677368 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.677386 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.677391 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.677461 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.677511 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.677527 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:48:02.677600 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.676497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:42927 (local address 127.18.96.130:46029)
0928 14:48:02.676541 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.676543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.676614 (+ 71us) server_negotiation.cc:407] Connection header received
0928 14:48:02.676683 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.676684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.676688 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.676742 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.677362 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.677363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.677363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.677368 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.677386 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.677391 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.677455 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.677511 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.677527 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:48:02.677631 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.676358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:35547 (local address 127.18.96.129:52533)
0928 14:48:02.676399 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.676410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:02.676451 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:02.676687 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:02.676687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:02.676690 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:02.677232 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:02.677232 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:02.677242 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:02.677257 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:02.677463 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:02.677463 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:02.677464 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:48:02.677513 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:48:02.677532 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 14:48:02.677659 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.676358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:46029 (local address 127.18.96.129:42927)
0928 14:48:02.676552 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.676560 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:02.676611 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:02.676745 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:02.676746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:02.676747 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:02.677243 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:02.677243 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:02.677247 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:02.677257 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:02.677460 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:02.677461 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:02.677461 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 14:48:02.677513 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:48:02.677533 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":52,"thread_start_us":86,"threads_started":1}
I20260928 14:48:02.678022 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d8c36e229247ac90c080521793ea4d" candidate_uuid: "7d77867f58c3452eb2433eb54586c15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "073424376b7246f9b4e18071e118e2b3" is_pre_election: true
I20260928 14:48:02.678037 23698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d8c36e229247ac90c080521793ea4d" candidate_uuid: "7d77867f58c3452eb2433eb54586c15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a0db6193804fd286d2c86b456c9025" is_pre_election: true
I20260928 14:48:02.678198 23829 raft_consensus.cc:2454] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d77867f58c3452eb2433eb54586c15c in term 0.
I20260928 14:48:02.678197 23698 raft_consensus.cc:2454] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d77867f58c3452eb2433eb54586c15c in term 0.
I20260928 14:48:02.678453 23500 leader_election.cc:302] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 073424376b7246f9b4e18071e118e2b3, 7d77867f58c3452eb2433eb54586c15c; no voters:
I20260928 14:48:02.678591 23896 raft_consensus.cc:2781] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:02.678645 23896 raft_consensus.cc:482] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:02.678674 23896 raft_consensus.cc:3037] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:02.679375 23896 raft_consensus.cc:504] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } }
I20260928 14:48:02.679524 23896 leader_election.cc:288] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [CANDIDATE]: Term 1 election: Requested vote from peers 073424376b7246f9b4e18071e118e2b3 (127.18.96.131:35547), 24a0db6193804fd286d2c86b456c9025 (127.18.96.130:46029)
I20260928 14:48:02.679656 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d8c36e229247ac90c080521793ea4d" candidate_uuid: "7d77867f58c3452eb2433eb54586c15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "073424376b7246f9b4e18071e118e2b3"
I20260928 14:48:02.679677 23698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d8c36e229247ac90c080521793ea4d" candidate_uuid: "7d77867f58c3452eb2433eb54586c15c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a0db6193804fd286d2c86b456c9025"
I20260928 14:48:02.679741 23698 raft_consensus.cc:3037] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:02.679735 23829 raft_consensus.cc:3037] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:02.680665 23829 raft_consensus.cc:2454] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d77867f58c3452eb2433eb54586c15c in term 1.
I20260928 14:48:02.680665 23698 raft_consensus.cc:2454] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d77867f58c3452eb2433eb54586c15c in term 1.
I20260928 14:48:02.680905 23501 leader_election.cc:302] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a0db6193804fd286d2c86b456c9025, 7d77867f58c3452eb2433eb54586c15c; no voters:
I20260928 14:48:02.681018 23896 raft_consensus.cc:2781] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:02.681185 23896 raft_consensus.cc:686] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [term 1 LEADER]: Becoming Leader. State: Replica: 7d77867f58c3452eb2433eb54586c15c, State: Running, Role: LEADER
I20260928 14:48:02.681269 23896 consensus_queue.cc:237] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } }
I20260928 14:48:02.682312 23424 catalog_manager.cc:5495] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d77867f58c3452eb2433eb54586c15c (127.18.96.129). New cstate: current_term: 1 leader_uuid: "7d77867f58c3452eb2433eb54586c15c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d77867f58c3452eb2433eb54586c15c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 } health_report { overall_health: UNKNOWN } } }
I20260928 14:48:02.687443 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.686460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.18.96.129:34393)
0928 14:48:02.686526 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.686527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.686536 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:02.686594 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.686595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.686605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.686656 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.687183 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.687184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.687184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.687188 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.687202 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.687206 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.687257 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.687297 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.687309 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:48:02.689162 23829 raft_consensus.cc:1261] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [term 1 FOLLOWER]: Refusing update from remote peer 7d77867f58c3452eb2433eb54586c15c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:02.689162 23698 raft_consensus.cc:1261] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [term 1 FOLLOWER]: Refusing update from remote peer 7d77867f58c3452eb2433eb54586c15c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:02.689538 23899 consensus_queue.cc:1035] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a0db6193804fd286d2c86b456c9025" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:02.689642 23896 consensus_queue.cc:1035] T 79d8c36e229247ac90c080521793ea4d P 7d77867f58c3452eb2433eb54586c15c [LEADER]: Connected to new peer: Peer: permanent_uuid: "073424376b7246f9b4e18071e118e2b3" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:48:02.690295 23876 tablet.cc:2369] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:02.692970 23904 mvcc.cc:205] Tried to move back new op lower bound from 7334325791492268032 to 7334325791463387136. Current Snapshot: MvccSnapshot[applied={T|T < 7334325791492268032 or (T in {7334325791492268032})}]
I20260928 14:48:02.693907 23424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:02.694003 23424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:02.694758 23424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb998d2547b84720b2531eeeb6e3538c: Sending DeleteTablet for 3 replicas of tablet 79d8c36e229247ac90c080521793ea4d
I20260928 14:48:02.695099 23809 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d8c36e229247ac90c080521793ea4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:02 UTC) from {username='slave'} at 127.0.0.1:47522
I20260928 14:48:02.695101 23547 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d8c36e229247ac90c080521793ea4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:02 UTC) from {username='slave'} at 127.0.0.1:44290
I20260928 14:48:02.695313 23678 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d8c36e229247ac90c080521793ea4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:02 UTC) from {username='slave'} at 127.0.0.1:43526
I20260928 14:48:02.695503 23914 tablet_replica.cc:330] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: stopping tablet replica
I20260928 14:48:02.695603 23914 raft_consensus.cc:2229] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:02.695710 23914 raft_consensus.cc:2258] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:02.695837 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 23485
I20260928 14:48:02.695719 23913 tablet_replica.cc:330] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: stopping tablet replica
I20260928 14:48:02.696762 23913 raft_consensus.cc:2229] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:02.696877 23913 raft_consensus.cc:2258] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:02.697173 23914 ts_tablet_manager.cc:1905] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:02.698621 23913 ts_tablet_manager.cc:1905] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:02.698726 23914 ts_tablet_manager.cc:1918] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:02.698817 23914 log.cc:1196] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/79d8c36e229247ac90c080521793ea4d
I20260928 14:48:02.699077 23914 ts_tablet_manager.cc:1939] T 79d8c36e229247ac90c080521793ea4d P 24a0db6193804fd286d2c86b456c9025: Deleting consensus metadata
I20260928 14:48:02.699532 23410 catalog_manager.cc:4841] TS 24a0db6193804fd286d2c86b456c9025 (127.18.96.130:46029): tablet 79d8c36e229247ac90c080521793ea4d (table test-table [id=96e53fb2e5474768b1f052788552abf4]) successfully deleted
I20260928 14:48:02.700250 23913 ts_tablet_manager.cc:1918] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:02.700342 23913 log.cc:1196] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/79d8c36e229247ac90c080521793ea4d
I20260928 14:48:02.700595 23913 ts_tablet_manager.cc:1939] T 79d8c36e229247ac90c080521793ea4d P 073424376b7246f9b4e18071e118e2b3: Deleting consensus metadata
I20260928 14:48:02.700999 23409 catalog_manager.cc:4841] TS 073424376b7246f9b4e18071e118e2b3 (127.18.96.131:35547): tablet 79d8c36e229247ac90c080521793ea4d (table test-table [id=96e53fb2e5474768b1f052788552abf4]) successfully deleted
W20260928 14:48:02.702136 23409 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:34393 (error 108)
I20260928 14:48:02.702361 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 23616
I20260928 14:48:02.702919 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.702469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:34393 (local address 127.0.0.1:44310)
0928 14:48:02.702842 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.702880 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:34393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
W20260928 14:48:02.703364 23409 catalog_manager.cc:4584] TS 7d77867f58c3452eb2433eb54586c15c (127.18.96.129:34393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79d8c36e229247ac90c080521793ea4d: Network error: Client connection negotiation failed: client connection to 127.18.96.129:34393: connect: Connection refused (error 111)
I20260928 14:48:02.707216 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 23747
I20260928 14:48:02.712189 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 23394
2026-09-28T14:48:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (788 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 14:48:02.732633 18818 mini_oidc.cc:152] Starting JWKS server
I20260928 14:48:02.733721 18818 webserver.cc:460] Webserver started at https://127.0.0.1:43125/ using document root <none> and password file <none>
I20260928 14:48:02.733795 18818 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:48:02.733995 18818 webserver.cc:460] Webserver started at http://127.0.0.1:36303/ using document root <none> and password file <none>
2026-09-28T14:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:02Z Disabled control of system clock
I20260928 14:48:02.742100 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:37735
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:37735
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43125/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/testchainca.cert with env {}
W20260928 14:48:02.824188 23921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:02.824410 23921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:02.824433 23921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:02.826016 23921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:02.826081 23921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:02.826095 23921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:02.826109 23921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:02.826121 23921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:48:02.826154 23921 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:48:02.826181 23921 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43125/jwks.json
I20260928 14:48:02.827821 23921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:37735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:37735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43125/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:02.828140 23921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:02.828390 23921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:02.833549 23927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.833693 23921 server_base.cc:1034] running on GCE node
W20260928 14:48:02.833561 23926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:02.833725 23929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:02.834036 23921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:02.834347 23921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:02.835557 23921 hybrid_clock.cc:648] HybridClock initialized: now 1790606882835538 us; error 27 us; skew 500 ppm
I20260928 14:48:02.836920 23921 webserver.cc:460] Webserver started at http://127.18.96.190:40925/ using document root <none> and password file <none>
I20260928 14:48:02.837167 23921 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:02.837237 23921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:02.837396 23921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:02.838470 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "5ccdfca4767344aa8bc18f698aa0d7ab"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "f08f805709713e938c7b18c5ffafa2a7"
server_key_iv: "fa6e90bf31c6d4156df7e8a07c760b5d"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.838850 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "5ccdfca4767344aa8bc18f698aa0d7ab"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "f08f805709713e938c7b18c5ffafa2a7"
server_key_iv: "fa6e90bf31c6d4156df7e8a07c760b5d"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.840273 23921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:02.841079 23935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:02.841255 23921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:48:02.841362 23921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "5ccdfca4767344aa8bc18f698aa0d7ab"
format_stamp: "Formatted at 2026-09-28 14:48:02 on dist-test-slave-j968"
server_key: "f08f805709713e938c7b18c5ffafa2a7"
server_key_iv: "fa6e90bf31c6d4156df7e8a07c760b5d"
server_key_version: "encryptionkey@0"
I20260928 14:48:02.841437 23921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:02.854377 23921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:02.854714 23921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:02.854869 23921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:02.858820 23921 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:37735
I20260928 14:48:02.858932 23987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:37735 every 8 connection(s)
I20260928 14:48:02.859239 23921 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:02.860513 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:02.862876 23988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab: Bootstrap starting.
I20260928 14:48:02.863492 23988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:02.863795 23988 log.cc:824] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:02.864537 23988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab: No bootstrap required, opened a new log
I20260928 14:48:02.866683 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 23921
I20260928 14:48:02.866806 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:02.866808 23988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccdfca4767344aa8bc18f698aa0d7ab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 37735 } }
I20260928 14:48:02.866981 18818 external_mini_cluster.cc:1428] Setting key daa5aa7d235b14b9a65132efd585888d
I20260928 14:48:02.866993 23988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:02.867020 23988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccdfca4767344aa8bc18f698aa0d7ab, State: Initialized, Role: FOLLOWER
I20260928 14:48:02.867553 23988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccdfca4767344aa8bc18f698aa0d7ab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 37735 } }
I20260928 14:48:02.867666 23988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:02.867712 23988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:02.867763 23988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:02.868474 23988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccdfca4767344aa8bc18f698aa0d7ab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 37735 } }
I20260928 14:48:02.868561 23988 leader_election.cc:302] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ccdfca4767344aa8bc18f698aa0d7ab; no voters:
I20260928 14:48:02.868747 23988 leader_election.cc:288] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:02.868780 23993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:02.868886 23993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [term 1 LEADER]: Becoming Leader. State: Replica: 5ccdfca4767344aa8bc18f698aa0d7ab, State: Running, Role: LEADER
I20260928 14:48:02.868968 23993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccdfca4767344aa8bc18f698aa0d7ab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 37735 } }
I20260928 14:48:02.869011 23988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:02.869304 23993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ccdfca4767344aa8bc18f698aa0d7ab. Latest consensus state: current_term: 1 leader_uuid: "5ccdfca4767344aa8bc18f698aa0d7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccdfca4767344aa8bc18f698aa0d7ab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 37735 } } }
I20260928 14:48:02.869392 23993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:02.869313 23995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ccdfca4767344aa8bc18f698aa0d7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccdfca4767344aa8bc18f698aa0d7ab" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 37735 } } }
I20260928 14:48:02.869549 23995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:02.869608 23998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:02.870148 23998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:02.871824 23998 catalog_manager.cc:1347] Generated new cluster ID: d65b434423044296aff67658486ce8dc
I20260928 14:48:02.871906 23998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:02.873449 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.867464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.18.96.190:37735)
0928 14:48:02.867657 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.867666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.867699 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:48:02.867830 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.867837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.867928 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.868063 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:02.868303 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:02.869961 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:02.871197 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:02.871791 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:02.872270 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:02.872743 (+ 473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:02.872747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:02.872747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:02.872947 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:02.872988 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:02.873002 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:02.873068 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:02.873168 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:48:02.873243 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260928 14:48:02.874553 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:48:02.877336 23998 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:02.878024 23998 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:02.884683 23998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ccdfca4767344aa8bc18f698aa0d7ab: Generated new TSK 0
I20260928 14:48:02.884868 23998 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:48:02.908013 24018 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 14:48:02.908140 24018 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:48:02.908381 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:02.892156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.18.96.190:37735)
0928 14:48:02.892220 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:02.892222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:02.892370 (+ 148us) server_negotiation.cc:407] Connection header received
0928 14:48:02.892476 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:02.892477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:02.892488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:02.892539 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:48:02.892692 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:02.893189 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:02.894584 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:02.894762 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:02.894815 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:02.895042 (+ 227us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:48:02.908130 (+ 13088us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:48:02.908325 (+ 195us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:48:02.908439 23992 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 14:48:02.909164 24017 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:02.892098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:37735 (local address 127.0.0.1:38038)
0928 14:48:02.892296 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:02.892308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:02.892380 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:02.892554 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:02.892554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:02.892556 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:48:02.892629 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:02.892631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:02.893483 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:02.893484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:02.894491 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:02.894494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:02.894847 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:02.894848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:02.894914 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:02.894938 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:48:02.908586 (+ 13648us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:48:02.908776 (+ 190us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.96.190:37735: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.18.96.190:37735: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 14:48:02.909732 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 23921
2026-09-28T14:48:02Z chronyd exiting
I20260928 14:48:02.945945 18818 test_util.cc:175] -----------------------------------------------
I20260928 14:48:02.946034 18818 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606857360589-18818-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 14:48:02.946749 18818 mini_oidc.cc:152] Starting JWKS server
I20260928 14:48:02.947458 18818 webserver.cc:460] Webserver started at https://127.0.0.1:44373/ using document root <none> and password file <none>
I20260928 14:48:02.947499 18818 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:48:02.947614 18818 webserver.cc:460] Webserver started at http://127.0.0.1:46503/ using document root <none> and password file <none>
2026-09-28T14:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:02Z Disabled control of system clock
I20260928 14:48:02.955175 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34435
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:34435
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44373/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/testchainca.cert with env {}
W20260928 14:48:03.039217 24024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:03.039490 24024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:03.039533 24024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:03.040906 24024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:03.040990 24024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:03.041011 24024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:03.041033 24024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:03.041064 24024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:48:03.041090 24024 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:48:03.041111 24024 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44373/jwks.json
I20260928 14:48:03.042799 24024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:34435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44373/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:03.043107 24024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:03.043336 24024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:03.048321 24032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:03.048321 24029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:03.048405 24030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:03.048540 24024 server_base.cc:1034] running on GCE node
I20260928 14:48:03.048780 24024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:03.049084 24024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:03.050282 24024 hybrid_clock.cc:648] HybridClock initialized: now 1790606883050271 us; error 40 us; skew 500 ppm
I20260928 14:48:03.051574 24024 webserver.cc:460] Webserver started at http://127.18.96.190:33911/ using document root <none> and password file <none>
I20260928 14:48:03.051819 24024 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:03.051914 24024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:03.052074 24024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:03.053186 24024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "3da3a3d7911d494eabe17f69f10eb30b"
format_stamp: "Formatted at 2026-09-28 14:48:03 on dist-test-slave-j968"
server_key: "3a18ae59a0fbcbc92a730b70b659efea"
server_key_iv: "a1803dffe5333bdae40cee6fe86194b8"
server_key_version: "encryptionkey@0"
I20260928 14:48:03.053560 24024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "3da3a3d7911d494eabe17f69f10eb30b"
format_stamp: "Formatted at 2026-09-28 14:48:03 on dist-test-slave-j968"
server_key: "3a18ae59a0fbcbc92a730b70b659efea"
server_key_iv: "a1803dffe5333bdae40cee6fe86194b8"
server_key_version: "encryptionkey@0"
I20260928 14:48:03.054873 24024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:03.055594 24038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:03.055764 24024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:03.055863 24024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "3da3a3d7911d494eabe17f69f10eb30b"
format_stamp: "Formatted at 2026-09-28 14:48:03 on dist-test-slave-j968"
server_key: "3a18ae59a0fbcbc92a730b70b659efea"
server_key_iv: "a1803dffe5333bdae40cee6fe86194b8"
server_key_version: "encryptionkey@0"
I20260928 14:48:03.055986 24024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:03.068409 24024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:03.068706 24024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:03.068856 24024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:03.072779 24024 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:34435
I20260928 14:48:03.072856 24090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:34435 every 8 connection(s)
I20260928 14:48:03.073220 24024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:03.073871 24091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:03.076602 24091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b: Bootstrap starting.
I20260928 14:48:03.077157 24091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:03.077399 24091 log.cc:824] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:03.078085 24091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b: No bootstrap required, opened a new log
I20260928 14:48:03.079456 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 24024
I20260928 14:48:03.079572 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:03.079756 18818 external_mini_cluster.cc:1428] Setting key 103284738ad1e1e30059215a9c73c5c0
I20260928 14:48:03.080009 24091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da3a3d7911d494eabe17f69f10eb30b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34435 } }
I20260928 14:48:03.080215 24091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:03.080259 24091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da3a3d7911d494eabe17f69f10eb30b, State: Initialized, Role: FOLLOWER
I20260928 14:48:03.080372 24091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da3a3d7911d494eabe17f69f10eb30b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34435 } }
I20260928 14:48:03.080459 24091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:03.080505 24091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:03.080561 24091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:03.081271 24091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da3a3d7911d494eabe17f69f10eb30b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34435 } }
I20260928 14:48:03.081362 24091 leader_election.cc:302] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da3a3d7911d494eabe17f69f10eb30b; no voters:
I20260928 14:48:03.081543 24091 leader_election.cc:288] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:03.081791 24096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:03.081808 24091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:03.082029 24096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [term 1 LEADER]: Becoming Leader. State: Replica: 3da3a3d7911d494eabe17f69f10eb30b, State: Running, Role: LEADER
I20260928 14:48:03.082129 24096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da3a3d7911d494eabe17f69f10eb30b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34435 } }
I20260928 14:48:03.082477 24097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da3a3d7911d494eabe17f69f10eb30b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da3a3d7911d494eabe17f69f10eb30b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34435 } } }
I20260928 14:48:03.082579 24097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:03.082724 24098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da3a3d7911d494eabe17f69f10eb30b. Latest consensus state: current_term: 1 leader_uuid: "3da3a3d7911d494eabe17f69f10eb30b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da3a3d7911d494eabe17f69f10eb30b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34435 } } }
I20260928 14:48:03.082772 24098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:03.082970 24105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:03.083700 24105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:03.085630 24105 catalog_manager.cc:1347] Generated new cluster ID: daaf47b8498343dd841878ebff9d1520
I20260928 14:48:03.085682 24105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:03.085882 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:03.080301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.18.96.190:34435)
0928 14:48:03.080550 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:03.080561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:03.080594 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:48:03.080710 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:03.080716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:03.080768 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:03.080880 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:03.081118 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.082514 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.083686 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.084294 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.084419 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:03.085145 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:03.085148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:03.085149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:03.085366 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:03.085412 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:03.085428 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:03.085498 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:03.085602 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:48:03.085683 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260928 14:48:03.086808 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:48:03.090885 24105 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:03.091571 24105 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:03.094520 24105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da3a3d7911d494eabe17f69f10eb30b: Generated new TSK 0
I20260928 14:48:03.094676 24105 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:48:03.122991 24121 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 14:48:03.123090 24121 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:48:03.123454 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:03.100808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.18.96.190:34435)
0928 14:48:03.100854 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:03.100856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:03.100966 (+ 110us) server_negotiation.cc:407] Connection header received
0928 14:48:03.101039 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:03.101040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:03.101050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:03.101101 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:48:03.101235 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.101700 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.102908 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.103112 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.103166 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:03.103359 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:48:03.123208 (+ 19849us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:48:03.123396 (+ 188us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36760: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 14:48:03.123538 24095 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36760: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 14:48:03.123584 24120 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:03.100711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34435 (local address 127.0.0.1:36760)
0928 14:48:03.100880 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:03.100892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:03.100965 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:03.101111 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:03.101112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:03.101114 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:48:03.101174 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:03.101176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.101797 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.101798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:03.102823 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:03.102826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.103190 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.103191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:03.103257 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:03.103264 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:48:03.123359 (+ 20095us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:48:03.123483 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.96.190:34435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260928 14:48:03.123972 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 24024
2026-09-28T14:48:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (215 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 14:48:03.162077 18818 mini_oidc.cc:152] Starting JWKS server
I20260928 14:48:03.162806 18818 webserver.cc:460] Webserver started at https://127.0.0.1:40225/ using document root <none> and password file <none>
I20260928 14:48:03.162891 18818 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:48:03.163079 18818 webserver.cc:460] Webserver started at http://127.0.0.1:38275/ using document root <none> and password file <none>
2026-09-28T14:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:03Z Disabled control of system clock
I20260928 14:48:03.170639 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:33281
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:33281
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40225/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:48:03.253329 24127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:03.253592 24127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:03.253631 24127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:03.254793 24127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:03.254863 24127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:03.254884 24127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:03.254915 24127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:03.254964 24127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:48:03.254994 24127 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:48:03.255020 24127 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40225/jwks.json
I20260928 14:48:03.256466 24127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:33281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40225/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:03.256748 24127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:03.257045 24127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:03.261960 24135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:03.261968 24132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:03.261978 24133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:03.261998 24127 server_base.cc:1034] running on GCE node
I20260928 14:48:03.262531 24127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:03.262825 24127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:03.263984 24127 hybrid_clock.cc:648] HybridClock initialized: now 1790606883263974 us; error 44 us; skew 500 ppm
I20260928 14:48:03.265221 24127 webserver.cc:460] Webserver started at http://127.18.96.190:42711/ using document root <none> and password file <none>
I20260928 14:48:03.265467 24127 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:03.265548 24127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:03.265713 24127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:03.266799 24127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "69f6fe3a344e4358b3f2259b2818611d"
format_stamp: "Formatted at 2026-09-28 14:48:03 on dist-test-slave-j968"
server_key: "c00acb6c9fe15ee1aaf68e39385e1030"
server_key_iv: "9cb2202fa08db8fd4e76354a0f566999"
server_key_version: "encryptionkey@0"
I20260928 14:48:03.267179 24127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "69f6fe3a344e4358b3f2259b2818611d"
format_stamp: "Formatted at 2026-09-28 14:48:03 on dist-test-slave-j968"
server_key: "c00acb6c9fe15ee1aaf68e39385e1030"
server_key_iv: "9cb2202fa08db8fd4e76354a0f566999"
server_key_version: "encryptionkey@0"
I20260928 14:48:03.268607 24127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:03.269373 24141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:03.269585 24127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:03.269683 24127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "69f6fe3a344e4358b3f2259b2818611d"
format_stamp: "Formatted at 2026-09-28 14:48:03 on dist-test-slave-j968"
server_key: "c00acb6c9fe15ee1aaf68e39385e1030"
server_key_iv: "9cb2202fa08db8fd4e76354a0f566999"
server_key_version: "encryptionkey@0"
I20260928 14:48:03.269764 24127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:03.283978 24127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:03.284337 24127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:03.284500 24127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:03.288707 24127 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:33281
I20260928 14:48:03.288741 24193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:33281 every 8 connection(s)
I20260928 14:48:03.289129 24127 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:03.289855 24194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:03.292742 24194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d: Bootstrap starting.
I20260928 14:48:03.293279 24194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:03.293521 24194 log.cc:824] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:03.294193 24194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d: No bootstrap required, opened a new log
I20260928 14:48:03.294945 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 24127
I20260928 14:48:03.295071 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:03.295251 18818 external_mini_cluster.cc:1428] Setting key ea20e146b5cb74cb80dca41312743a1a
I20260928 14:48:03.296304 24194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f6fe3a344e4358b3f2259b2818611d" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33281 } }
I20260928 14:48:03.296476 24194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:03.296512 24194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f6fe3a344e4358b3f2259b2818611d, State: Initialized, Role: FOLLOWER
I20260928 14:48:03.296681 24194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f6fe3a344e4358b3f2259b2818611d" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33281 } }
I20260928 14:48:03.296772 24194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:03.296804 24194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:03.296854 24194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:03.297610 24194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f6fe3a344e4358b3f2259b2818611d" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33281 } }
I20260928 14:48:03.297720 24194 leader_election.cc:302] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f6fe3a344e4358b3f2259b2818611d; no voters:
I20260928 14:48:03.297865 24194 leader_election.cc:288] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:03.297914 24199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:03.298092 24199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [term 1 LEADER]: Becoming Leader. State: Replica: 69f6fe3a344e4358b3f2259b2818611d, State: Running, Role: LEADER
I20260928 14:48:03.298110 24194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:03.298187 24199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f6fe3a344e4358b3f2259b2818611d" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33281 } }
I20260928 14:48:03.298553 24200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69f6fe3a344e4358b3f2259b2818611d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f6fe3a344e4358b3f2259b2818611d" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33281 } } }
I20260928 14:48:03.298542 24199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f6fe3a344e4358b3f2259b2818611d. Latest consensus state: current_term: 1 leader_uuid: "69f6fe3a344e4358b3f2259b2818611d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f6fe3a344e4358b3f2259b2818611d" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33281 } } }
I20260928 14:48:03.298702 24200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:03.298713 24199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:03.299085 24208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:03.299612 24208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:03.302026 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:03.295785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.18.96.190:33281)
0928 14:48:03.296071 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:03.296080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:03.296118 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:48:03.296264 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:03.296271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:03.296355 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:03.296508 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:03.296720 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.298557 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.299788 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.300420 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.300512 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:03.301247 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:03.301251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:03.301251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:03.301430 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:03.301465 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:03.301477 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:03.301533 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:03.301634 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 14:48:03.301740 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":156,"threads_started":1}
I20260928 14:48:03.302980 24208 catalog_manager.cc:1347] Generated new cluster ID: 1c8af54ff9ab48d399c0a4d284e6c9dd
I20260928 14:48:03.302978 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:48:03.303037 24208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:03.317232 24208 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:03.317848 24208 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:03.323618 24208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f6fe3a344e4358b3f2259b2818611d: Generated new TSK 0
I20260928 14:48:03.323839 24208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:48:03.334874 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:03.331133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.18.96.190:33281)
0928 14:48:03.331176 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:03.331180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:03.331232 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:48:03.331324 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:03.331325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:03.331334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:03.331395 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:48:03.331539 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.332109 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.333270 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.333455 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.333511 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:03.333701 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:48:03.334653 (+ 952us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:48:03.334788 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 14:48:03.334942 24198 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 14:48:03.335259 24223 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:03.330979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:33281 (local address 127.0.0.1:52440)
0928 14:48:03.331168 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:03.331181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:03.331253 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:03.331412 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:03.331413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:03.331415 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:48:03.331478 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:03.331480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.332191 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.332192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:03.333182 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:03.333184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.333536 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.333537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:03.333601 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:03.333607 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:48:03.334759 (+ 1152us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:48:03.334871 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.96.190:33281: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40225/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
I20260928 14:48:03.335427 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 24127
E20260928 14:48:03.346088 24224 webserver.cc:533] Webserver: sslize failed:
E20260928 14:48:03.346168 24224 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T14:48:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T14:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:03Z Disabled control of system clock
I20260928 14:48:03.380679 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34115
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:34115
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/testchainca.cert with env {}
W20260928 14:48:03.462603 24228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:03.462828 24228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:03.462852 24228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:03.464032 24228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:03.464076 24228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:03.464089 24228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:03.464102 24228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:03.464114 24228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:48:03.464126 24228 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:48:03.464138 24228 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 14:48:03.465621 24228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:34115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:03.465821 24228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:03.465998 24228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:03.471378 24233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:03.471393 24236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:03.471388 24234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:03.471513 24228 server_base.cc:1034] running on GCE node
I20260928 14:48:03.472138 24228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:03.472545 24228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:03.473711 24228 hybrid_clock.cc:648] HybridClock initialized: now 1790606883473690 us; error 33 us; skew 500 ppm
I20260928 14:48:03.475229 24228 webserver.cc:460] Webserver started at http://127.18.96.190:41191/ using document root <none> and password file <none>
I20260928 14:48:03.475502 24228 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:03.475579 24228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:03.475729 24228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:03.476899 24228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557"
format_stamp: "Formatted at 2026-09-28 14:48:03 on dist-test-slave-j968"
server_key: "76549b1bcb512cf99fa09cf46f9da65b"
server_key_iv: "6c31d0d70d83c1be3af8e27369e27778"
server_key_version: "encryptionkey@0"
I20260928 14:48:03.477283 24228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557"
format_stamp: "Formatted at 2026-09-28 14:48:03 on dist-test-slave-j968"
server_key: "76549b1bcb512cf99fa09cf46f9da65b"
server_key_iv: "6c31d0d70d83c1be3af8e27369e27778"
server_key_version: "encryptionkey@0"
I20260928 14:48:03.478610 24228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:48:03.479460 24242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:03.479627 24228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:03.479736 24228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557"
format_stamp: "Formatted at 2026-09-28 14:48:03 on dist-test-slave-j968"
server_key: "76549b1bcb512cf99fa09cf46f9da65b"
server_key_iv: "6c31d0d70d83c1be3af8e27369e27778"
server_key_version: "encryptionkey@0"
I20260928 14:48:03.479813 24228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:03.490617 24228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:03.490948 24228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:03.491101 24228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:03.495049 24228 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:34115
I20260928 14:48:03.495087 24294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:34115 every 8 connection(s)
I20260928 14:48:03.495450 24228 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:03.496426 24295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:03.498639 24295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557: Bootstrap starting.
I20260928 14:48:03.499234 24295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:03.499531 24295 log.cc:824] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:03.500394 24295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557: No bootstrap required, opened a new log
I20260928 14:48:03.502250 24295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34115 } }
I20260928 14:48:03.502386 24295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:03.502408 24295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ad6cfcb8e74ded9ab4fbbea9d55557, State: Initialized, Role: FOLLOWER
I20260928 14:48:03.502573 24295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34115 } }
I20260928 14:48:03.502636 24295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:03.502658 24295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:03.502686 24295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:03.505057 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 24228
I20260928 14:48:03.505148 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:03.505293 18818 external_mini_cluster.cc:1428] Setting key 5c7eb131e17b06d3b58ab6de45b78c71
I20260928 14:48:03.510650 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:03.505783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.18.96.190:34115)
0928 14:48:03.506095 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:03.506103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:03.506128 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:48:03.506237 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:03.506243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:03.506295 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:03.506443 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:03.506688 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.507829 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.508722 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.509155 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.509289 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:03.510016 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:03.510020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:03.510020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:03.510197 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:03.510229 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:03.510239 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:03.510311 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:03.510396 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 14:48:03.510479 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":140,"threads_started":1}
I20260928 14:48:03.511708 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:48:03.576578 24295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34115 } }
I20260928 14:48:03.576787 24295 leader_election.cc:302] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69ad6cfcb8e74ded9ab4fbbea9d55557; no voters:
I20260928 14:48:03.577051 24295 leader_election.cc:288] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:03.577139 24300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:03.577320 24300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [term 1 LEADER]: Becoming Leader. State: Replica: 69ad6cfcb8e74ded9ab4fbbea9d55557, State: Running, Role: LEADER
I20260928 14:48:03.577389 24295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:03.577466 24300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34115 } }
I20260928 14:48:03.577986 24301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34115 } } }
I20260928 14:48:03.578073 24301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:03.578295 24302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69ad6cfcb8e74ded9ab4fbbea9d55557. Latest consensus state: current_term: 1 leader_uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad6cfcb8e74ded9ab4fbbea9d55557" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34115 } } }
I20260928 14:48:03.578639 24302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:03.578852 24309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:03.579540 24309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:03.581321 24309 catalog_manager.cc:1347] Generated new cluster ID: 002e0f4d4c5c44dca54d3b79ef1ca8ee
I20260928 14:48:03.581394 24309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:03.594221 24309 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:03.594844 24309 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:03.598445 24309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69ad6cfcb8e74ded9ab4fbbea9d55557: Generated new TSK 0
I20260928 14:48:03.598656 24309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:48:04.047227 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:03.614683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.18.96.190:34115)
0928 14:48:03.614747 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:03.614749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:03.614840 (+ 91us) server_negotiation.cc:407] Connection header received
0928 14:48:03.615017 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:03.615019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:03.615030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:03.615110 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:48:03.615269 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.615775 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.616634 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.616837 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.616930 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:03.617116 (+ 186us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:48:04.046784 (+429668us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:48:04.047136 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46946: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 14:48:04.047305 24324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:03.614606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34115 (local address 127.0.0.1:46946)
0928 14:48:03.614752 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:03.614768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:03.614862 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:03.615123 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:03.615125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:03.615127 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:48:03.615195 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:03.615197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.615905 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.615906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:03.616540 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:03.616542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:03.616934 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:03.616935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:03.617000 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:03.617004 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:48:04.047065 (+430061us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:48:04.047193 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.96.190:34115: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
W20260928 14:48:04.047322 24299 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46946: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260928 14:48:04.047688 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 24228
2026-09-28T14:48:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (692 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (98 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (98 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:04 dist-test-slave-j968 krb5kdc[24336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:04 dist-test-slave-j968 krb5kdc[24336](info): set up 2 sockets
Sep 28 14:48:04 dist-test-slave-j968 krb5kdc[24336](info): commencing operation
krb5kdc: starting...
W20260928 14:48:05.293066 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:48:05 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:48:05.419607 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:33569
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:33569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:05.499995 24349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:05.500253 24349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:05.500289 24349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:05.501470 24349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:05.501549 24349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:05.501572 24349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:05.501616 24349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:05.501644 24349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:05.503127 24349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:33569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:33569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:05.503409 24349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:05.503640 24349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:05.508507 24355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:05.508507 24349 server_base.cc:1034] running on GCE node
W20260928 14:48:05.508507 24354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:05.508533 24357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:05.508993 24349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:05.509289 24349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:05.510494 24349 hybrid_clock.cc:648] HybridClock initialized: now 1790606885510476 us; error 27 us; skew 500 ppm
Sep 28 14:48:05 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606885, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:05.513401 24349 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:48:05.513828 24349 webserver.cc:460] Webserver started at http://127.18.96.190:40901/ using document root <none> and password file <none>
I20260928 14:48:05.514072 24349 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:05.514118 24349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:05.514218 24349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:05.515349 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "af3b3a784ba447a2b84d6c75e65930f0"
format_stamp: "Formatted at 2026-09-28 14:48:05 on dist-test-slave-j968"
server_key: "5f91fc517dd88f93870c5ac0084bb82d"
server_key_iv: "487810827b6359de5dcc202d60d48fd4"
server_key_version: "encryptionkey@0"
I20260928 14:48:05.515689 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "af3b3a784ba447a2b84d6c75e65930f0"
format_stamp: "Formatted at 2026-09-28 14:48:05 on dist-test-slave-j968"
server_key: "5f91fc517dd88f93870c5ac0084bb82d"
server_key_iv: "487810827b6359de5dcc202d60d48fd4"
server_key_version: "encryptionkey@0"
I20260928 14:48:05.517433 24349 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:48:05.518566 24364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:05.518779 24349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:05.518884 24349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "af3b3a784ba447a2b84d6c75e65930f0"
format_stamp: "Formatted at 2026-09-28 14:48:05 on dist-test-slave-j968"
server_key: "5f91fc517dd88f93870c5ac0084bb82d"
server_key_iv: "487810827b6359de5dcc202d60d48fd4"
server_key_version: "encryptionkey@0"
I20260928 14:48:05.519047 24349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:05.555816 24349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:05.558060 24349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:05.558192 24349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:05.562582 24349 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:33569
I20260928 14:48:05.562672 24416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:33569 every 8 connection(s)
I20260928 14:48:05.563087 24349 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:05.564123 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:05.564862 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 24349
I20260928 14:48:05.564996 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:05.565238 18818 external_mini_cluster.cc:1428] Setting key 75bbd67b57f2a5b9ad2670ea22619207
I20260928 14:48:05.567359 24417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0: Bootstrap starting.
I20260928 14:48:05.568053 24417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:05.568343 24417 log.cc:824] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:05.569032 24417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0: No bootstrap required, opened a new log
I20260928 14:48:05.570964 24417 raft_consensus.cc:348] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3b3a784ba447a2b84d6c75e65930f0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33569 } }
I20260928 14:48:05.571123 24417 raft_consensus.cc:374] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:05.571171 24417 raft_consensus.cc:729] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3b3a784ba447a2b84d6c75e65930f0, State: Initialized, Role: FOLLOWER
I20260928 14:48:05.571331 24417 consensus_queue.cc:260] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3b3a784ba447a2b84d6c75e65930f0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33569 } }
I20260928 14:48:05.571416 24417 raft_consensus.cc:388] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:05.571460 24417 raft_consensus.cc:482] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:05.571507 24417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:05.572105 24417 raft_consensus.cc:504] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3b3a784ba447a2b84d6c75e65930f0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33569 } }
I20260928 14:48:05.572219 24417 leader_election.cc:302] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af3b3a784ba447a2b84d6c75e65930f0; no voters:
I20260928 14:48:05.572405 24417 leader_election.cc:288] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:05.572438 24422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 14:48:05 dist-test-slave-j968 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:05.572602 24422 raft_consensus.cc:686] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [term 1 LEADER]: Becoming Leader. State: Replica: af3b3a784ba447a2b84d6c75e65930f0, State: Running, Role: LEADER
I20260928 14:48:05.572727 24417 sys_catalog.cc:564] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:05.572691 24422 consensus_queue.cc:237] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3b3a784ba447a2b84d6c75e65930f0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33569 } }
I20260928 14:48:05.573267 24422 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af3b3a784ba447a2b84d6c75e65930f0. Latest consensus state: current_term: 1 leader_uuid: "af3b3a784ba447a2b84d6c75e65930f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3b3a784ba447a2b84d6c75e65930f0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33569 } } }
I20260928 14:48:05.573542 24422 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:05.573467 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af3b3a784ba447a2b84d6c75e65930f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3b3a784ba447a2b84d6c75e65930f0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 33569 } } }
I20260928 14:48:05.573805 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0 [sys.catalog]: This master's current role is: LEADER
W20260928 14:48:05.574697 24437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:48:05.574754 24437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:48:05.575620 24431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:05.575832 24431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:05.577010 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:05.565819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.18.96.190:33569)
0928 14:48:05.566026 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:05.566036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:05.566070 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:48:05.566617 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:05.566626 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:05.566694 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:05.566844 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:05.567528 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.568704 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.569578 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.570061 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.570205 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:05.573232 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:05.573236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:05.573236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:05.573250 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:05.575345 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:05.575699 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:05.575702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:05.575703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:05.575817 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:05.576103 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:05.576104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:05.576105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:05.576344 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:05.576461 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:05.576670 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 14:48:05.576809 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
I20260928 14:48:05.577713 24431 catalog_manager.cc:1347] Generated new cluster ID: bd511e05f80e489e952d55683504ec82
I20260928 14:48:05.577767 24431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:05.593531 24431 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:05.594331 24431 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:05.613065 24431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af3b3a784ba447a2b84d6c75e65930f0: Generated new TSK 0
I20260928 14:48:05.613323 24431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:48:05.646983 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33569
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:05.734642 24443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:05.734859 24443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:05.734884 24443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:05.736110 24443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:05.736157 24443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:05.736186 24443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:05.737668 24443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:05.737859 24443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:05.738050 24443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:05.743342 24449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:05.743360 24448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:05.743393 24443 server_base.cc:1034] running on GCE node
W20260928 14:48:05.743342 24451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:05.744007 24443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:05.744266 24443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:05.745431 24443 hybrid_clock.cc:648] HybridClock initialized: now 1790606885745410 us; error 36 us; skew 500 ppm
Sep 28 14:48:05 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606885, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:05.748073 24443 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:48:05.748476 24443 webserver.cc:460] Webserver started at http://127.18.96.129:39077/ using document root <none> and password file <none>
I20260928 14:48:05.748662 24443 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:05.748701 24443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:05.748793 24443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:05.749814 24443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "07fb20f03f9f44afb299e333e47d4c45"
format_stamp: "Formatted at 2026-09-28 14:48:05 on dist-test-slave-j968"
server_key: "9e88e0920527e4634f9eb3f3fcc75520"
server_key_iv: "9d3299c0729577aff44e3d5acf794a01"
server_key_version: "encryptionkey@0"
I20260928 14:48:05.750114 24443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "07fb20f03f9f44afb299e333e47d4c45"
format_stamp: "Formatted at 2026-09-28 14:48:05 on dist-test-slave-j968"
server_key: "9e88e0920527e4634f9eb3f3fcc75520"
server_key_iv: "9d3299c0729577aff44e3d5acf794a01"
server_key_version: "encryptionkey@0"
I20260928 14:48:05.751394 24443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:05.752279 24458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:05.752456 24443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:48:05.752552 24443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "07fb20f03f9f44afb299e333e47d4c45"
format_stamp: "Formatted at 2026-09-28 14:48:05 on dist-test-slave-j968"
server_key: "9e88e0920527e4634f9eb3f3fcc75520"
server_key_iv: "9d3299c0729577aff44e3d5acf794a01"
server_key_version: "encryptionkey@0"
I20260928 14:48:05.752617 24443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:05.770255 24443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:05.772886 24443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:05.773005 24443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:05.773208 24443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:05.773541 24443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:05.773576 24443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:05.773636 24443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:05.773664 24443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:05.779462 24443 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:37713
I20260928 14:48:05.779556 24571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:37713 every 8 connection(s)
I20260928 14:48:05.779946 24443 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:05.782347 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 24443
I20260928 14:48:05.782526 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:05.782744 18818 external_mini_cluster.cc:1428] Setting key b4a2cab82f0dce4965b499d9d6ed7f0a
Sep 28 14:48:05 dist-test-slave-j968 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606885, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:05.790560 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:05.780691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:38619 (local address 127.18.96.190:33569)
0928 14:48:05.780740 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:05.780742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:05.781012 (+ 270us) server_negotiation.cc:407] Connection header received
0928 14:48:05.782009 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:05.782010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:05.782024 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:05.782078 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:05.782841 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.783479 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.784709 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.784934 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.785019 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:05.787451 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:05.787453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:05.787453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:05.787461 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:05.789199 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:05.789579 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:05.789580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:05.789581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:05.789650 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:05.789884 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:05.789885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:05.789885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:05.790041 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:05.790118 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:05.790352 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 14:48:05.790452 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:48:05.790587 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:05.780686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:33569 (local address 127.18.96.129:38619)
0928 14:48:05.780883 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:05.780921 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:05.781883 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:05.782149 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:05.782154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:05.782442 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:05.782747 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:05.782752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.783573 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.783574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:05.784602 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:05.784605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.785037 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.785038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:05.785164 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:05.785698 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:05.785706 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:05.787339 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:05.789280 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:05.789286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:05.789287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:05.789504 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:05.789722 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:05.789723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:05.789723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:05.789817 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:05.790111 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:05.790115 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:05.790224 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 14:48:05.790303 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:48:05.790419 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
I20260928 14:48:05.791136 24572 heartbeater.cc:347] Connected to a master server at 127.18.96.190:33569
I20260928 14:48:05.791249 24572 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:05.791467 24572 heartbeater.cc:511] Master 127.18.96.190:33569 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
I20260928 14:48:05.792136 24381 ts_manager.cc:194] Registered new tserver with Master: 07fb20f03f9f44afb299e333e47d4c45 (127.18.96.129:37713)
I20260928 14:48:05.792774 24381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:38619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:48:05.810341 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33569
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:05.892354 24577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:05.892565 24577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:05.892588 24577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:05.893775 24577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:05.893822 24577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:05.893849 24577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:05.895358 24577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:05.895602 24577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:05.895854 24577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:05.901041 24583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:05.901100 24582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:05.901115 24585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:05.901314 24577 server_base.cc:1034] running on GCE node
I20260928 14:48:05.901520 24577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:05.901801 24577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:05.903002 24577 hybrid_clock.cc:648] HybridClock initialized: now 1790606885902979 us; error 39 us; skew 500 ppm
Sep 28 14:48:05 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606885, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:05.905774 24577 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:48:05.906333 24577 webserver.cc:460] Webserver started at http://127.18.96.130:36597/ using document root <none> and password file <none>
I20260928 14:48:05.906581 24577 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:05.906652 24577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:05.906800 24577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:05.908017 24577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "9fb5be73fbf64ad4a3a2cb0c531f875b"
format_stamp: "Formatted at 2026-09-28 14:48:05 on dist-test-slave-j968"
server_key: "69b858b94ff13fc5c37e0e7ce82616b8"
server_key_iv: "7690aaf152da48ede1c8013280073a8a"
server_key_version: "encryptionkey@0"
I20260928 14:48:05.908380 24577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "9fb5be73fbf64ad4a3a2cb0c531f875b"
format_stamp: "Formatted at 2026-09-28 14:48:05 on dist-test-slave-j968"
server_key: "69b858b94ff13fc5c37e0e7ce82616b8"
server_key_iv: "7690aaf152da48ede1c8013280073a8a"
server_key_version: "encryptionkey@0"
I20260928 14:48:05.909741 24577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:48:05.910595 24592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:05.910818 24577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:05.910924 24577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "9fb5be73fbf64ad4a3a2cb0c531f875b"
format_stamp: "Formatted at 2026-09-28 14:48:05 on dist-test-slave-j968"
server_key: "69b858b94ff13fc5c37e0e7ce82616b8"
server_key_iv: "7690aaf152da48ede1c8013280073a8a"
server_key_version: "encryptionkey@0"
I20260928 14:48:05.911031 24577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:05.935563 24577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:05.937817 24577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:05.938027 24577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:05.938287 24577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:05.938627 24577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:05.938681 24577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:05.938726 24577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:05.938751 24577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:05.944854 24577 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:34567
I20260928 14:48:05.944911 24705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:34567 every 8 connection(s)
I20260928 14:48:05.945223 24577 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:05.945590 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 24577
I20260928 14:48:05.945719 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:05.945974 18818 external_mini_cluster.cc:1428] Setting key 4392729365db15efe9542456c20c3c92
Sep 28 14:48:05 dist-test-slave-j968 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606885, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
I20260928 14:48:05.956766 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:05.946043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:47955 (local address 127.18.96.190:33569)
0928 14:48:05.946093 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:05.946096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:05.946299 (+ 203us) server_negotiation.cc:407] Connection header received
0928 14:48:05.947857 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:05.947858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:05.947914 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:05.947984 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:05.948768 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.949530 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.951173 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.951355 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.951409 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:05.953870 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:05.953871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:05.953872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:05.953880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:05.955293 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:05.955754 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:05.955755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:05.955756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:05.955815 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:05.956081 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:05.956082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:05.956082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:05.956202 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:05.956279 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:05.956583 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:48:05.956670 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:48:05.956943 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:05.945934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:33569 (local address 127.18.96.130:47955)
0928 14:48:05.946151 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:05.946206 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:05.947742 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:05.948066 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:05.948071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:05.948458 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:05.948653 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:05.948657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.949622 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.949624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:05.951089 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:05.951092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:05.951440 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:05.951441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:05.951614 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:05.952188 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:05.952199 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:05.953771 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:05.955395 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:05.955402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:05.955404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:05.955665 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:05.955891 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:05.955892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:05.955893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:05.956009 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:05.956270 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:05.956276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:05.956421 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 14:48:05.956540 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 14:48:05.956701 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
I20260928 14:48:05.957432 24706 heartbeater.cc:347] Connected to a master server at 127.18.96.190:33569
I20260928 14:48:05.957552 24706 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
I20260928 14:48:05.957804 24706 heartbeater.cc:511] Master 127.18.96.190:33569 requested a full tablet report, sending...
I20260928 14:48:05.958222 24381 ts_manager.cc:194] Registered new tserver with Master: 9fb5be73fbf64ad4a3a2cb0c531f875b (127.18.96.130:34567)
I20260928 14:48:05.958617 24381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:47955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:48:05.976958 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33569
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:06.064832 24711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:06.065037 24711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:06.065059 24711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:06.066206 24711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:06.066246 24711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:06.066298 24711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:06.067751 24711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:33569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:06.068017 24711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:06.068255 24711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:06.073302 24716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:06.073406 24717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:06.073297 24719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:06.073597 24711 server_base.cc:1034] running on GCE node
I20260928 14:48:06.073822 24711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:06.074072 24711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:06.075258 24711 hybrid_clock.cc:648] HybridClock initialized: now 1790606886075247 us; error 27 us; skew 500 ppm
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:06.077786 24711 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:48:06.078225 24711 webserver.cc:460] Webserver started at http://127.18.96.131:43601/ using document root <none> and password file <none>
I20260928 14:48:06.078466 24711 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:06.078536 24711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:06.078692 24711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:06.079746 24711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "175c881ae8564c5ea5b87487bef459fd"
format_stamp: "Formatted at 2026-09-28 14:48:06 on dist-test-slave-j968"
server_key: "4b1b8be629f7c2809d26eda6c3078ce8"
server_key_iv: "fbd608a42815bf91c10a4fb04f7d9bf8"
server_key_version: "encryptionkey@0"
I20260928 14:48:06.080211 24711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "175c881ae8564c5ea5b87487bef459fd"
format_stamp: "Formatted at 2026-09-28 14:48:06 on dist-test-slave-j968"
server_key: "4b1b8be629f7c2809d26eda6c3078ce8"
server_key_iv: "fbd608a42815bf91c10a4fb04f7d9bf8"
server_key_version: "encryptionkey@0"
I20260928 14:48:06.081602 24711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:06.082304 24726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:06.082469 24711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:06.082566 24711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "175c881ae8564c5ea5b87487bef459fd"
format_stamp: "Formatted at 2026-09-28 14:48:06 on dist-test-slave-j968"
server_key: "4b1b8be629f7c2809d26eda6c3078ce8"
server_key_iv: "fbd608a42815bf91c10a4fb04f7d9bf8"
server_key_version: "encryptionkey@0"
I20260928 14:48:06.082664 24711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:06.103757 24711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:06.106104 24711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:06.106302 24711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:06.106586 24711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:06.106915 24711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:06.106981 24711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:06.107023 24711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:06.107050 24711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:06.113255 24711 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:43607
I20260928 14:48:06.113314 24839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:43607 every 8 connection(s)
I20260928 14:48:06.113667 24711 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:06.122593 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 24711
I20260928 14:48:06.122711 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:06.122928 18818 external_mini_cluster.cc:1428] Setting key 6131a1cc03dde8aab70cc78ce92da6c2
I20260928 14:48:06.124194 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:06.114513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:53793 (local address 127.18.96.190:33569)
0928 14:48:06.114564 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:06.114569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:06.114758 (+ 189us) server_negotiation.cc:407] Connection header received
0928 14:48:06.115790 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:06.115791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:06.115802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:06.115949 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:06.116621 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.117137 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.118219 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.118426 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.118493 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:06.120965 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:06.120967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:06.120967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:06.120975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:06.122489 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:06.123028 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:06.123030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:06.123030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:06.123102 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:06.123447 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:06.123448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:06.123449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:06.123626 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:06.123689 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:06.124053 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 14:48:06.124127 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:48:06.124372 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:06.114448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:33569 (local address 127.18.96.131:53793)
0928 14:48:06.114618 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:06.114662 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:06.115682 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:06.116028 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:06.116032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:06.116290 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:06.116492 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:06.116496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.117268 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.117269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:06.118110 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:06.118112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.118512 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.118513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:06.118631 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:06.119175 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:06.119183 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:06.120856 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:06.122570 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:06.122576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:06.122588 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:06.122914 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:06.123173 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:06.123174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:06.123174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:06.123377 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:06.123705 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:06.123710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:06.123905 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 14:48:06.124017 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:48:06.124159 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260928 14:48:06.124822 24840 heartbeater.cc:347] Connected to a master server at 127.18.96.190:33569
I20260928 14:48:06.124922 24840 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:06.125089 24840 heartbeater.cc:511] Master 127.18.96.190:33569 requested a full tablet report, sending...
I20260928 14:48:06.125483 24381 ts_manager.cc:194] Registered new tserver with Master: 175c881ae8564c5ea5b87487bef459fd (127.18.96.131:43607)
I20260928 14:48:06.125871 24381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:53793
I20260928 14:48:06.134974 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/test-admin.keytab.
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:06.152486 18818 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 14:48:06.154186 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:06.153429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.18.96.129:37713)
0928 14:48:06.153612 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:06.153616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:06.153626 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:06.153653 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:06.153654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:06.153669 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:06.153733 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:06.154126 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49226: BlockingRecv error: recv got EOF from 127.0.0.1:49226 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
W20260928 14:48:06.154315 18818 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:06.155429 18818 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:06.156582 18818 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:48:06.164987 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:06.157239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.18.96.129:37713)
0928 14:48:06.157317 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:06.157319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:06.157329 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:06.157373 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:06.157374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:06.157386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:06.157432 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:06.157853 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.158676 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.159427 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.159751 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.159811 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:06.161974 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:06.161982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:06.161982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:06.161988 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:06.163687 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:06.164154 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:06.164158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:06.164158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:06.164223 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:06.164401 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:06.164402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:06.164402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:06.164549 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:06.164674 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:06.164713 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 14:48:06.164779 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 14:48:06.166997 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:06.166327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.18.96.129:37713)
0928 14:48:06.166386 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:06.166388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:06.166498 (+ 110us) server_negotiation.cc:407] Connection header received
0928 14:48:06.166559 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:06.166559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:06.166571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:06.166642 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:06.166916 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49250: BlockingRecv error: recv got EOF from 127.0.0.1:49250 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:48:06.167145 18818 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:06.168183 18818 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:06.169144 18818 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:48:06.176151 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:06.169693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.18.96.129:37713)
0928 14:48:06.169739 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:06.169740 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:06.169762 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:48:06.169809 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:06.169810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:06.169817 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:06.169869 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:06.170171 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.170670 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.171333 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.171529 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.171583 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:06.173885 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:06.173887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:06.173887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:06.173893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:06.175283 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:06.175560 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:06.175561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:06.175561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:06.175608 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:06.175748 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:06.175748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:06.175748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:06.175824 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:06.175946 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:06.176030 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 14:48:06.176092 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 14:48:06.177673 24847 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:06.177176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.18.96.129:37713)
0928 14:48:06.177252 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:06.177253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:06.177283 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:48:06.177330 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:06.177330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:06.177337 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:06.177380 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:06.177641 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49256: BlockingRecv error: recv got EOF from 127.0.0.1:49256 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 14:48:06.177752 18818 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606857360589-18818-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:06.178701 18818 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:06.179543 18818 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:48:06.186264 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:06.180033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.18.96.129:37713)
0928 14:48:06.180109 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:06.180112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:06.180194 (+ 82us) server_negotiation.cc:407] Connection header received
0928 14:48:06.180264 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:06.180265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:06.180274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:06.180328 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:06.180685 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.181135 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.181799 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:06.182003 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:06.182088 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:06.184016 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:06.184018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:06.184018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:06.184026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:06.185192 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:06.185527 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:06.185528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:06.185529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:06.185600 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:06.185799 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:06.185800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:06.185800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:06.185898 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:06.185994 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:06.186097 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 14:48:06.186169 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:48:06.186841 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 24443
I20260928 14:48:06.192286 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 24577
I20260928 14:48:06.197077 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 24711
I20260928 14:48:06.201766 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 24349
2026-09-28T14:48:06Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1958 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24860](info): set up 2 sockets
Sep 28 14:48:06 dist-test-slave-j968 krb5kdc[24860](info): commencing operation
krb5kdc: starting...
W20260928 14:48:07.309269 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:48:07 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.190.keytab.
Entry for principal oryx/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.190.keytab.
I20260928 14:48:07.431344 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:39113
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.190.keytab
--principal=oryx/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:39113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:07.513842 24873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:07.514113 24873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:07.514148 24873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:07.515295 24873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:07.515367 24873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:07.515391 24873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:07.515411 24873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:07.515435 24873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:07.517017 24873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:39113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.190.keytab
--principal=oryx/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:07.517308 24873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:07.517544 24873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:07.522574 24881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:07.522636 24879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:07.522637 24878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:07.522763 24873 server_base.cc:1034] running on GCE node
I20260928 14:48:07.523096 24873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:07.523399 24873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:07.524595 24873 hybrid_clock.cc:648] HybridClock initialized: now 1790606887524586 us; error 35 us; skew 500 ppm
Sep 28 14:48:07 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:07.527336 24873 init.cc:377] Logged in from keytab as oryx/127.18.96.190@KRBTEST.COM (short username oryx)
I20260928 14:48:07.527791 24873 webserver.cc:460] Webserver started at http://127.18.96.190:41241/ using document root <none> and password file <none>
I20260928 14:48:07.528083 24873 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:07.528162 24873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:07.528321 24873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:07.529534 24873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "db5e569e3cd147088982678eeae71397"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "b2c9fc6b1aaa4e5f740d22ff8385ab3f"
server_key_iv: "63abefee04fea9ac3509af3d0623182e"
server_key_version: "encryptionkey@0"
I20260928 14:48:07.529910 24873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "db5e569e3cd147088982678eeae71397"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "b2c9fc6b1aaa4e5f740d22ff8385ab3f"
server_key_iv: "63abefee04fea9ac3509af3d0623182e"
server_key_version: "encryptionkey@0"
I20260928 14:48:07.531353 24873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:07.532330 24888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:07.532536 24873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:48:07.532668 24873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "db5e569e3cd147088982678eeae71397"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "b2c9fc6b1aaa4e5f740d22ff8385ab3f"
server_key_iv: "63abefee04fea9ac3509af3d0623182e"
server_key_version: "encryptionkey@0"
I20260928 14:48:07.532755 24873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:07.558828 24873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:07.561362 24873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:07.561543 24873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:07.565809 24873 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:39113
I20260928 14:48:07.565848 24940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:39113 every 8 connection(s)
I20260928 14:48:07.566236 24873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:07.566519 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 24873
I20260928 14:48:07.566655 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:07.566931 18818 external_mini_cluster.cc:1428] Setting key 98e3d641308064755e2708d5a9af8115
I20260928 14:48:07.567448 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:07.570777 24941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: Bootstrap starting.
I20260928 14:48:07.571429 24941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:07.571753 24941 log.cc:824] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:07.572527 24941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: No bootstrap required, opened a new log
Sep 28 14:48:07 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:07.574630 24941 raft_consensus.cc:348] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } }
I20260928 14:48:07.574782 24941 raft_consensus.cc:374] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:07.574807 24941 raft_consensus.cc:729] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5e569e3cd147088982678eeae71397, State: Initialized, Role: FOLLOWER
I20260928 14:48:07.574954 24941 consensus_queue.cc:260] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } }
I20260928 14:48:07.575066 24941 raft_consensus.cc:388] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:07.575114 24941 raft_consensus.cc:482] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:07.575173 24941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:07.575914 24941 raft_consensus.cc:504] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } }
I20260928 14:48:07.576032 24941 leader_election.cc:302] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db5e569e3cd147088982678eeae71397; no voters:
I20260928 14:48:07.576213 24941 leader_election.cc:288] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:07.576316 24946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:07.576467 24946 raft_consensus.cc:686] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 1 LEADER]: Becoming Leader. State: Replica: db5e569e3cd147088982678eeae71397, State: Running, Role: LEADER
I20260928 14:48:07.576491 24941 sys_catalog.cc:564] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:07.576565 24946 consensus_queue.cc:237] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } }
I20260928 14:48:07.576866 24946 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db5e569e3cd147088982678eeae71397. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } }
I20260928 14:48:07.576968 24946 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:07.576877 24948 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } }
I20260928 14:48:07.577049 24948 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
W20260928 14:48:07.578452 24962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:48:07.578506 24962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:48:07.578553 24954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:07.578706 24954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:07.579087 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:07.567537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.18.96.190:39113)
0928 14:48:07.567732 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:07.567741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:07.567775 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:48:07.568305 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:07.568314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:07.568380 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:07.568535 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:07.569190 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.570539 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.571405 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.572062 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.572149 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:07.575064 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:07.575067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:07.575068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:07.575080 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:07.577293 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:07.577703 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:07.577706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:07.577706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:07.577826 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:07.578039 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:07.578040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:07.578040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:07.578274 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:07.578432 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:07.578794 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 14:48:07.578939 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260928 14:48:07.580277 24954 catalog_manager.cc:1347] Generated new cluster ID: 0c27a911cfc5475a9bc64e29ac98a8b4
I20260928 14:48:07.580335 24954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:07.603598 24954 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:07.604245 24954 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:07.613525 24954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: Generated new TSK 0
I20260928 14:48:07.613744 24954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab.
Entry for principal oryx/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab.
I20260928 14:48:07.647614 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:39113
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab
--principal=oryx/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:07.729615 24967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:07.729871 24967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:07.729910 24967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:07.731061 24967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:07.731138 24967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:07.731185 24967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:07.732754 24967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab
--principal=oryx/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:39113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:07.733033 24967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:07.733386 24967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:07.739074 24975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:07.739055 24973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:07.739295 24967 server_base.cc:1034] running on GCE node
W20260928 14:48:07.739305 24972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:07.739620 24967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:07.739905 24967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:07.741077 24967 hybrid_clock.cc:648] HybridClock initialized: now 1790606887741063 us; error 25 us; skew 500 ppm
Sep 28 14:48:07 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:07.743737 24967 init.cc:377] Logged in from keytab as oryx/127.18.96.129@KRBTEST.COM (short username oryx)
I20260928 14:48:07.744277 24967 webserver.cc:460] Webserver started at http://127.18.96.129:34777/ using document root <none> and password file <none>
I20260928 14:48:07.744513 24967 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:07.744581 24967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:07.744738 24967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:07.745792 24967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "02c3b535fb9541e89be93fcb8d8155c1"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "eba46694bdc94a1eaa3296fe8119c517"
server_key_iv: "5119ceacde5df536516aa9323c5bf8eb"
server_key_version: "encryptionkey@0"
I20260928 14:48:07.746170 24967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "02c3b535fb9541e89be93fcb8d8155c1"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "eba46694bdc94a1eaa3296fe8119c517"
server_key_iv: "5119ceacde5df536516aa9323c5bf8eb"
server_key_version: "encryptionkey@0"
I20260928 14:48:07.747491 24967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:07.748365 24982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:07.748539 24967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:07.748638 24967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "02c3b535fb9541e89be93fcb8d8155c1"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "eba46694bdc94a1eaa3296fe8119c517"
server_key_iv: "5119ceacde5df536516aa9323c5bf8eb"
server_key_version: "encryptionkey@0"
I20260928 14:48:07.748720 24967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:07.758577 24967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:07.760960 24967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:07.761123 24967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:07.761345 24967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:07.761658 24967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:07.761711 24967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:07.761750 24967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:07.761777 24967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:07.767660 24967 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:38549
I20260928 14:48:07.767730 25095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:38549 every 8 connection(s)
I20260928 14:48:07.768085 24967 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:07.772967 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 24967
I20260928 14:48:07.773084 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:07.773293 18818 external_mini_cluster.cc:1428] Setting key c18e4cbe97e360348018bcd4ab33ef3d
Sep 28 14:48:07 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:07.777943 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:07.768873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:53899 (local address 127.18.96.190:39113)
0928 14:48:07.768931 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:07.768934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:07.769201 (+ 267us) server_negotiation.cc:407] Connection header received
0928 14:48:07.769641 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:07.769642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:07.769652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:07.769724 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:07.770690 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.771193 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.772200 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.772400 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.772468 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:07.774904 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:07.774905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:07.774906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:07.774914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:07.776429 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:07.776841 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:07.776842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:07.776842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:07.776950 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:07.777240 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:07.777241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:07.777242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:07.777364 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:07.777398 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:07.777762 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 14:48:07.777853 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:48:07.778167 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:07.768863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.129:53899)
0928 14:48:07.769026 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:07.769063 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:07.769555 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:07.769782 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:07.769785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:07.770354 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:07.770550 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:07.770555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.771276 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.771277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:07.772127 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:07.772129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.772476 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.772477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:07.772578 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:07.773259 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:07.773267 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:07.774807 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:07.776519 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:07.776526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:07.776528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:07.776756 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:07.777017 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:07.777018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:07.777018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:07.777135 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:07.777483 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:07.777490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:07.777626 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 14:48:07.777728 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:48:07.777890 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260928 14:48:07.778726 25096 heartbeater.cc:347] Connected to a master server at 127.18.96.190:39113
I20260928 14:48:07.778810 25096 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:07.779017 25096 heartbeater.cc:511] Master 127.18.96.190:39113 requested a full tablet report, sending...
I20260928 14:48:07.779621 24905 ts_manager.cc:194] Registered new tserver with Master: 02c3b535fb9541e89be93fcb8d8155c1 (127.18.96.129:38549)
I20260928 14:48:07.780611 24905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:53899
WARNING: no policy specified for oryx/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab.
Entry for principal oryx/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab.
I20260928 14:48:07.799469 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:39113
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab
--principal=oryx/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:07.881021 25101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:07.881237 25101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:07.881259 25101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:07.882375 25101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:07.882412 25101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:07.882437 25101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:07.883852 25101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab
--principal=oryx/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:39113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:07.884126 25101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:07.884353 25101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:07.889334 25109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:07.889343 25107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:07.889536 25106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:07.889552 25101 server_base.cc:1034] running on GCE node
I20260928 14:48:07.889861 25101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:07.890094 25101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:07.891281 25101 hybrid_clock.cc:648] HybridClock initialized: now 1790606887891280 us; error 44 us; skew 500 ppm
Sep 28 14:48:07 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:07.893844 25101 init.cc:377] Logged in from keytab as oryx/127.18.96.130@KRBTEST.COM (short username oryx)
I20260928 14:48:07.894327 25101 webserver.cc:460] Webserver started at http://127.18.96.130:45529/ using document root <none> and password file <none>
I20260928 14:48:07.894569 25101 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:07.894645 25101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:07.894795 25101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:07.895838 25101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "2f6fc171f9f943a7bb6100700bc6c615"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "4e8f07ca9c6e6913e96eff9ac5b3c800"
server_key_iv: "9995dcb829a73409b7bb9d2740dc7edc"
server_key_version: "encryptionkey@0"
I20260928 14:48:07.896274 25101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "2f6fc171f9f943a7bb6100700bc6c615"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "4e8f07ca9c6e6913e96eff9ac5b3c800"
server_key_iv: "9995dcb829a73409b7bb9d2740dc7edc"
server_key_version: "encryptionkey@0"
I20260928 14:48:07.897589 25101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:07.898321 25116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:07.898509 25101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:07.898627 25101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "2f6fc171f9f943a7bb6100700bc6c615"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "4e8f07ca9c6e6913e96eff9ac5b3c800"
server_key_iv: "9995dcb829a73409b7bb9d2740dc7edc"
server_key_version: "encryptionkey@0"
I20260928 14:48:07.898710 25101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:07.918732 25101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:07.921473 25101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:07.921671 25101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:07.921928 25101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:07.922266 25101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:07.922322 25101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:07.922372 25101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:07.922402 25101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:07.928288 25101 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:44377
I20260928 14:48:07.928333 25229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:44377 every 8 connection(s)
I20260928 14:48:07.928668 25101 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:07.934583 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 25101
I20260928 14:48:07.934692 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:07.934931 18818 external_mini_cluster.cc:1428] Setting key 64a52de0b6444339c344d5b0ef99e22a
Sep 28 14:48:07 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:07.939152 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:07.929413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:55767 (local address 127.18.96.190:39113)
0928 14:48:07.929518 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:07.929521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:07.929688 (+ 167us) server_negotiation.cc:407] Connection header received
0928 14:48:07.930278 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:07.930279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:07.930292 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:07.930344 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:07.931326 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.931836 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.932883 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.933090 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.933175 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:07.935733 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:07.935735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:07.935735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:07.935743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:07.937538 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:07.938066 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:07.938067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:07.938067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:07.938121 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:07.938416 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:07.938417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:07.938417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:07.938582 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:07.938650 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:07.938919 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 14:48:07.939004 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:48:07.939189 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:07.929377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.130:55767)
0928 14:48:07.929565 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:07.929602 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:07.930155 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:07.930427 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:07.930430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:07.931021 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:07.931218 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:07.931222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.931981 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.931982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:07.932779 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:07.932781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:07.933186 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:07.933187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:07.933313 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:07.933801 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:07.933809 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:07.935639 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:07.937635 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:07.937642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:07.937644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:07.937973 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:07.938189 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:07.938189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:07.938190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:07.938343 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:07.938651 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:07.938656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:07.938784 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 14:48:07.938876 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:48:07.939003 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260928 14:48:07.940394 25230 heartbeater.cc:347] Connected to a master server at 127.18.96.190:39113
I20260928 14:48:07.940485 25230 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:07.940683 25230 heartbeater.cc:511] Master 127.18.96.190:39113 requested a full tablet report, sending...
I20260928 14:48:07.941104 24905 ts_manager.cc:194] Registered new tserver with Master: 2f6fc171f9f943a7bb6100700bc6c615 (127.18.96.130:44377)
I20260928 14:48:07.941473 24905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:55767
WARNING: no policy specified for oryx/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab.
Entry for principal oryx/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab.
I20260928 14:48:07.963493 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:39113
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab
--principal=oryx/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:08.046351 25235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:08.046571 25235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:08.046594 25235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:08.047784 25235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:08.047827 25235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:08.047906 25235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:08.049412 25235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab
--principal=oryx/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:39113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:08.049664 25235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:08.049901 25235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:08.055269 25240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.055248 25243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.055285 25241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.055631 25235 server_base.cc:1034] running on GCE node
I20260928 14:48:08.055898 25235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:08.056180 25235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:08.057318 25235 hybrid_clock.cc:648] HybridClock initialized: now 1790606888057301 us; error 39 us; skew 500 ppm
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:08.059837 25235 init.cc:377] Logged in from keytab as oryx/127.18.96.131@KRBTEST.COM (short username oryx)
I20260928 14:48:08.060335 25235 webserver.cc:460] Webserver started at http://127.18.96.131:38311/ using document root <none> and password file <none>
I20260928 14:48:08.060518 25235 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:08.060559 25235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:08.060647 25235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:08.061635 25235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "c4bbac146304448287d481482d4cc6bb"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "8cdb76ad99f038661c7e6faec2a8ec3b"
server_key_iv: "55f01e2de3d51cb37591cb07e9d47a72"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.061947 25235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "c4bbac146304448287d481482d4cc6bb"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "8cdb76ad99f038661c7e6faec2a8ec3b"
server_key_iv: "55f01e2de3d51cb37591cb07e9d47a72"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.063297 25235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:08.063977 25250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.064152 25235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:08.064250 25235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "c4bbac146304448287d481482d4cc6bb"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "8cdb76ad99f038661c7e6faec2a8ec3b"
server_key_iv: "55f01e2de3d51cb37591cb07e9d47a72"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.064337 25235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:08.078039 25235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:08.080099 25235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:08.080255 25235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:08.080503 25235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:08.080822 25235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:08.080876 25235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.080919 25235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:08.080947 25235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.086889 25235 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:40059
I20260928 14:48:08.086927 25363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:40059 every 8 connection(s)
I20260928 14:48:08.087270 25235 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:08.088745 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 25235
I20260928 14:48:08.088831 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:08.089020 18818 external_mini_cluster.cc:1428] Setting key a6f15c87b3da124c36544584e882c611
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:08.096375 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.087977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:44455 (local address 127.18.96.190:39113)
0928 14:48:08.088071 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.088074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.088267 (+ 193us) server_negotiation.cc:407] Connection header received
0928 14:48:08.088756 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.088758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.088771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.088858 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.090088 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.090564 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.091489 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.091680 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.091735 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.093831 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.093832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.093832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.093838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.095214 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.095572 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.095572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.095573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.095622 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.095823 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.095824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.095824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.095972 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.096023 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.096257 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.096312 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:48:08.096575 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.087983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.131:44455)
0928 14:48:08.088152 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.088194 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.088661 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.088969 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.088973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.089749 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.090006 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.090014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.090637 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.090638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.091412 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.091414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.091753 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.091754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.091848 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.092442 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.092459 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.093754 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.095288 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.095293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.095295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.095499 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.095678 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.095679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.095679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.095770 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.096028 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.096033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.096144 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.096230 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.096345 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 14:48:08.097043 25364 heartbeater.cc:347] Connected to a master server at 127.18.96.190:39113
I20260928 14:48:08.097155 25364 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:08.097325 25364 heartbeater.cc:511] Master 127.18.96.190:39113 requested a full tablet report, sending...
I20260928 14:48:08.097712 24905 ts_manager.cc:194] Registered new tserver with Master: c4bbac146304448287d481482d4cc6bb (127.18.96.131:40059)
I20260928 14:48:08.098093 24905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:44455
I20260928 14:48:08.100723 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.96.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.189.keytab.
Entry for principal oryx/127.18.96.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.189.keytab.
I20260928 14:48:08.127429 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.189:42805
--webserver_interface=127.18.96.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.189.keytab
--principal=oryx/127.18.96.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:39113,127.18.96.189:42805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:08.220866 25369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:08.221148 25369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:08.221176 25369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:08.222424 25369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:08.222466 25369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:08.222481 25369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:08.222494 25369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:08.222506 25369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:08.224056 25369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:39113,127.18.96.189:42805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.189.keytab
--principal=oryx/127.18.96.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.189:42805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.96.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
W20260928 14:48:08.224157 25369 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.96.190:39113,127.18.96.189:42805'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 14:48:08.224331 25369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:08.224560 25369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:08.229588 25374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.229588 25375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.229707 25369 server_base.cc:1034] running on GCE node
W20260928 14:48:08.229673 25377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.230041 25369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:08.230341 25369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:08.231511 25369 hybrid_clock.cc:648] HybridClock initialized: now 1790606888231485 us; error 38 us; skew 500 ppm
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:08.233999 25369 init.cc:377] Logged in from keytab as oryx/127.18.96.189@KRBTEST.COM (short username oryx)
I20260928 14:48:08.234436 25369 webserver.cc:460] Webserver started at http://127.18.96.189:33363/ using document root <none> and password file <none>
I20260928 14:48:08.234684 25369 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:08.234751 25369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:08.234915 25369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:08.236109 25369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data/instance:
uuid: "ec8f238e20044baf9d7ce74d06e053a2"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "eb75b6f87548f6a7eeb1569b3475ce04"
server_key_iv: "35ce481b5c2a0509e392566042b3d528"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.236471 25369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal/instance:
uuid: "ec8f238e20044baf9d7ce74d06e053a2"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "eb75b6f87548f6a7eeb1569b3475ce04"
server_key_iv: "35ce481b5c2a0509e392566042b3d528"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.237810 25369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:08.238564 25384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.238786 25369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:08.238909 25369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal
uuid: "ec8f238e20044baf9d7ce74d06e053a2"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "eb75b6f87548f6a7eeb1569b3475ce04"
server_key_iv: "35ce481b5c2a0509e392566042b3d528"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.239004 25369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:08.262795 25369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:08.264899 25369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:08.265085 25369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:08.269564 25369 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.189:42805
I20260928 14:48:08.269610 25436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.189:42805 every 8 connection(s)
I20260928 14:48:08.269989 25369 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
I20260928 14:48:08.270972 25437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:08.272625 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 25369
I20260928 14:48:08.272571 25437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.189@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.189@KRBTEST.COM
I20260928 14:48:08.282784 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.273316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.18.96.190:39113)
0928 14:48:08.273370 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.273372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.273613 (+ 241us) server_negotiation.cc:407] Connection header received
0928 14:48:08.274067 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.274068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.274081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.274149 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.275153 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.275911 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.276894 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.277168 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.277248 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.279597 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.279598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.279599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.279605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.281371 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.281776 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.281778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.281778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.281841 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.282105 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.282106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.282106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.282270 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.282349 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.282598 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.282681 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:48:08.282987 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.273301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:51870)
0928 14:48:08.273472 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.273513 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.273976 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.274743 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.274745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.274977 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.275060 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.275065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.275999 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.275999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.276798 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.276800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.277255 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.277256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.277361 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.277969 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.277979 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.279505 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.281451 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.281456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.281458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.281702 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.281923 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.281924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.281924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.282015 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.282340 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.282346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.282472 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.282597 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.282750 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260928 14:48:08.283560 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.273515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.18.96.189:42805)
0928 14:48:08.273660 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.273664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.273700 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:48:08.274039 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.274044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.274071 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.274161 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.274738 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.276069 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.276952 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.277329 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.277408 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.280185 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.280192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.280192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.280199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.281943 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.282276 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.282279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.282280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.282332 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.282541 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.282542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.282542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.282688 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.283152 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.283198 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.283436 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260928 14:48:08.283601 25437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.189@KRBTEST.COM for oryx/127.18.96.189@KRBTEST.COM
I20260928 14:48:08.289659 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.283982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.0.0.1:44050)
0928 14:48:08.284029 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.284039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.284072 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.284178 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.284179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.284323 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.284354 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.284355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.285162 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.285163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.285635 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.285637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.286044 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.286045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.286092 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.286459 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.286463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.287564 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.288888 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.288889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.288889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.289038 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.289204 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.289204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.289204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.289251 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.289420 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.289420 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.289451 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.289499 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.289567 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:48:08.289700 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.284035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.18.96.189:42805)
0928 14:48:08.284070 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.284072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.284081 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:08.284110 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.284111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.284121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.284175 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.284469 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.285062 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.285731 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.285963 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.286045 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.287654 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.287656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.287656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.287661 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.288821 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.289101 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.289101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.289102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.289146 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.289292 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.289292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.289293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.289373 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.289415 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.289513 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.289609 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":17}
I20260928 14:48:08.291026 25437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Bootstrap starting.
I20260928 14:48:08.291926 25437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:08.292217 25437 log.cc:824] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:08.293041 25437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: No bootstrap required, opened a new log
I20260928 14:48:08.294497 25437 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } }
I20260928 14:48:08.294618 25437 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:08.294639 25437 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8f238e20044baf9d7ce74d06e053a2, State: Initialized, Role: FOLLOWER
I20260928 14:48:08.294799 25437 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } }
I20260928 14:48:08.295147 25437 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:08.295104 25443 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } } }
I20260928 14:48:08.295222 25443 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:48:08.296916 25454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:48:08.297008 25454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:48:08.298667 25369 master_runner.cc:428] Detected that this master ec8f238e20044baf9d7ce74d06e053a2 is joining an existing cluster
I20260928 14:48:08.298735 25369 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.96.189:42805
I20260928 14:48:08.307427 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.299755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.18.96.190:39113)
0928 14:48:08.299937 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.299939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.299947 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:48:08.300005 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.300006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.300018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.300053 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.300611 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.301460 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.302369 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.302601 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.302692 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.303860 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.303861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.303862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.303908 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.305635 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.306487 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.306489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.306489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.306553 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.306761 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.306762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.306762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.306906 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.306979 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.307231 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.307312 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260928 14:48:08.307750 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.299930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.18.96.189:42805)
0928 14:48:08.299966 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.299968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.300112 (+ 144us) server_negotiation.cc:407] Connection header received
0928 14:48:08.300134 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.300135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.300144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.300197 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.301567 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.302291 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.303285 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.303596 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.303668 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.304982 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.304984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.304984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.304991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.306636 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.306953 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.306954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.306954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.307007 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.307199 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.307199 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.307200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.307322 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.307389 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.307561 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.307649 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:48:08.308068 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.299706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42330)
0928 14:48:08.299842 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.299853 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.299932 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.300123 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.300124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.300390 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.300525 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.300527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.301544 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.301545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.302284 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.302286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.302705 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.302706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.302769 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.303274 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.303280 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.303775 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.305962 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.305963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.305964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.306391 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.306632 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.306633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.306633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.306701 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.307055 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.307056 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.307104 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.307573 (+ 469us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.307704 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":52,"thread_start_us":84,"threads_started":1}
I20260928 14:48:08.308239 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.299762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.0.0.1:43442)
0928 14:48:08.300012 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.300022 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.300072 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.300212 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.300213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.300421 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.300525 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.300527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.302381 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.302381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.303212 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.303215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.303692 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.303693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.303768 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.304269 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.304273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.304645 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.306691 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.306692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.306692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.306882 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.307073 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.307073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.307074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.307135 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.307394 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.307394 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.307437 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.307917 (+ 480us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.307998 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.190@KRBTEST.COM for oryx/127.18.96.189@KRBTEST.COM
I20260928 14:48:08.318172 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.311368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.18.96.189:42805)
0928 14:48:08.311427 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.311429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.311455 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:48:08.311549 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.311550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.311561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.311611 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.312107 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.312556 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.313398 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.313587 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.313661 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.315630 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.315632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.315632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.315640 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.316951 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.317349 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.317351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.317351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.317400 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.317614 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.317615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.317615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.317723 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.317794 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.317990 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.318068 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:48:08.318178 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.311197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.0.0.1:43446)
0928 14:48:08.311341 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.311401 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.311482 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.311616 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.311620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.311970 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.312025 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.312027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.312638 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.312639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.313285 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.313287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.313671 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.313672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.313758 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.314217 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.314221 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.315539 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.317036 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.317044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.317046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.317255 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.317472 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.317472 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.317473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.317538 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.317800 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.317808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.317860 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.317952 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.318077 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260928 14:48:08.318861 24905 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.96.189:42805: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2"
member_type: NON_VOTER
last_known_addr {
host: "127.18.96.189"
port: 42805
}
attrs {
promote: true
}
}
dest_uuid: "db5e569e3cd147088982678eeae71397"
cas_config_opid_index: -1
I20260928 14:48:08.319232 24905 consensus_queue.cc:237] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } }
I20260928 14:48:08.319757 25464 raft_consensus.cc:2932] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ec8f238e20044baf9d7ce74d06e053a2 (127.18.96.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } } }
I20260928 14:48:08.320142 25411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:08.320894 25411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Refusing update from remote peer db5e569e3cd147088982678eeae71397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 14:48:08.321089 25464 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.96.189:42805
I20260928 14:48:08.321504 25463 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } } }
I20260928 14:48:08.321604 25463 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:08.321714 25369 master.cc:537] Master@127.18.96.189:42805 shutting down...
I20260928 14:48:08.322325 25463 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:48:08.323163 24891 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 14:48:08.323848 24891 consensus_peers.cc:487] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 -> Peer ec8f238e20044baf9d7ce74d06e053a2 (127.18.96.189:42805): Couldn't send request to peer ec8f238e20044baf9d7ce74d06e053a2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:48:08.326215 25369 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:08.326339 25369 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:08.326385 25369 tablet_replica.cc:330] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: stopping tablet replica
I20260928 14:48:08.340575 25369 master.cc:552] Master@127.18.96.189:42805 shutdown complete.
I20260928 14:48:08.340667 25369 master_runner.cc:305] Clearing existing system tablet
I20260928 14:48:08.340808 25369 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:08.340878 25369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:08.341652 25369 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:08.342350 25468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.342512 25369 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:08.342576 25369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:08.342619 25369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal
uuid: "ec8f238e20044baf9d7ce74d06e053a2"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "eb75b6f87548f6a7eeb1569b3475ce04"
server_key_iv: "35ce481b5c2a0509e392566042b3d528"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.343400 25369 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:08.345160 25369 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:48:08.345233 25369 log.cc:1196] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 14:48:08.345469 25369 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Deleting consensus metadata
I20260928 14:48:08.345613 25369 master_runner.cc:315] Copying system tablet from 127.18.96.190:39113
I20260928 14:48:08.346318 25369 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: tablet copy: Beginning tablet copy session from remote peer at address 127.18.96.190:39113
I20260928 14:48:08.353031 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.346742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.18.96.190:39113)
0928 14:48:08.346791 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.346794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.346988 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:48:08.347076 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.347077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.347088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.347147 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.347473 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.348068 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.348841 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.349057 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.349145 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.350218 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.350219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.350220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.350227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.351671 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.352202 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.352203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.352203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.352268 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.352470 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.352471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.352471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.352605 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.352691 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.352849 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.352932 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:48:08.353057 25474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.346672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42342)
0928 14:48:08.346844 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.346854 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.347005 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.347148 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.347149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.347331 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.347387 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.347389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.348148 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.348149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.348742 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.348743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.349147 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.349148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.349216 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.349734 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.349739 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.350103 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.351781 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.351782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.351782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.352101 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.352335 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.352335 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.352336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.352397 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.352682 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.352683 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.352730 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.352820 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.352933 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260928 14:48:08.353350 24925 tablet_copy_service.cc:140] P db5e569e3cd147088982678eeae71397: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ec8f238e20044baf9d7ce74d06e053a2 ({username='oryx', principal='oryx/127.18.96.189@KRBTEST.COM'} at 127.0.0.1:42342)
I20260928 14:48:08.353435 24925 tablet_copy_service.cc:161] P db5e569e3cd147088982678eeae71397: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ec8f238e20044baf9d7ce74d06e053a2 at {username='oryx', principal='oryx/127.18.96.189@KRBTEST.COM'} at 127.0.0.1:42342: session id = ec8f238e20044baf9d7ce74d06e053a2-00000000000000000000000000000000
I20260928 14:48:08.354130 24925 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:48:08.354764 25369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:48:08.355768 25369 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: tablet copy: Starting download of 0 data blocks...
I20260928 14:48:08.355959 25369 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: tablet copy: Starting download of 1 WAL segments...
I20260928 14:48:08.356902 25369 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:48:08.357779 24925 tablet_copy_service.cc:342] P db5e569e3cd147088982678eeae71397: Request end of tablet copy session ec8f238e20044baf9d7ce74d06e053a2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.96.189@KRBTEST.COM'} at 127.0.0.1:42342
I20260928 14:48:08.357841 24925 tablet_copy_service.cc:433] P db5e569e3cd147088982678eeae71397: ending tablet copy session ec8f238e20044baf9d7ce74d06e053a2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ec8f238e20044baf9d7ce74d06e053a2
I20260928 14:48:08.360321 25369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:08.361768 25475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.361788 25476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.365075 25478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.365237 25369 server_base.cc:1034] running on GCE node
I20260928 14:48:08.365348 25369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:08.365542 25369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:08.366686 25369 hybrid_clock.cc:648] HybridClock initialized: now 1790606888366676 us; error 29 us; skew 500 ppm
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:08.368233 25369 init.cc:377] Logged in from keytab as oryx/127.18.96.189@KRBTEST.COM (short username oryx)
I20260928 14:48:08.368536 25369 webserver.cc:460] Webserver started at http://127.18.96.189:37511/ using document root <none> and password file <none>
I20260928 14:48:08.368677 25369 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:08.368717 25369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:08.369370 25369 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:08.369954 25485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.370086 25369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:08.370159 25369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal
uuid: "ec8f238e20044baf9d7ce74d06e053a2"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "eb75b6f87548f6a7eeb1569b3475ce04"
server_key_iv: "35ce481b5c2a0509e392566042b3d528"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.370253 25369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:08.376194 25369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:08.378048 25369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:08.381186 25369 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.189:42805
I20260928 14:48:08.381201 25537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.189:42805 every 8 connection(s)
I20260928 14:48:08.381529 25369 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
I20260928 14:48:08.382565 25538 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:48:08.383176 25538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Bootstrap starting.
I20260928 14:48:08.386081 25538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:48:08.386297 25538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Bootstrap complete.
I20260928 14:48:08.386518 25538 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } }
I20260928 14:48:08.386610 25538 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec8f238e20044baf9d7ce74d06e053a2, State: Initialized, Role: LEARNER
I20260928 14:48:08.386647 25538 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } }
I20260928 14:48:08.386818 25538 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:08.386814 25542 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } } }
I20260928 14:48:08.387048 25542 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:48:08.388427 25553 catalog_manager.cc:1259] Loaded cluster ID: 0c27a911cfc5475a9bc64e29ac98a8b4
I20260928 14:48:08.388481 25553 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: loading cluster ID for follower catalog manager: success
I20260928 14:48:08.389580 25553 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: acquiring CA information for follower catalog manager: success
I20260928 14:48:08.389930 25553 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:48:08.393013 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.385363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.18.96.189:42805)
0928 14:48:08.385512 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.385514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.385520 (+ 6us) server_negotiation.cc:407] Connection header received
0928 14:48:08.385539 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.385540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.385548 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.385625 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.386171 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.387153 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.388134 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.388427 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.388503 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.390173 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.390174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.390175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.390186 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.391780 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.392199 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.392200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.392200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.392267 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.392494 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.392495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.392495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.392616 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.392727 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.392824 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.392916 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.18.96.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.188.keytab.
Entry for principal oryx/127.18.96.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.188.keytab.
I20260928 14:48:08.420331 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.188:37387
--webserver_interface=127.18.96.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.188.keytab
--principal=oryx/127.18.96.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:39113,127.18.96.189:42805,127.18.96.188:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:08.506788 25556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:08.507022 25556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:08.507047 25556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:08.508411 25556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:08.508481 25556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:08.508496 25556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:08.508509 25556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:08.508522 25556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:08.510130 25556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:39113,127.18.96.189:42805,127.18.96.188:37387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.188.keytab
--principal=oryx/127.18.96.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.188:37387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.96.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:08.510406 25556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:08.510676 25556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:08.515856 25561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.515902 25564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.515978 25556 server_base.cc:1034] running on GCE node
W20260928 14:48:08.515914 25562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.516326 25556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:08.516574 25556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:08.517717 25556 hybrid_clock.cc:648] HybridClock initialized: now 1790606888517703 us; error 28 us; skew 500 ppm
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:08.520501 25556 init.cc:377] Logged in from keytab as oryx/127.18.96.188@KRBTEST.COM (short username oryx)
I20260928 14:48:08.521014 25556 webserver.cc:460] Webserver started at http://127.18.96.188:33951/ using document root <none> and password file <none>
I20260928 14:48:08.521283 25556 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:08.521361 25556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:08.521653 25556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:08.522912 25556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data/instance:
uuid: "a2445c83ebe6491aacbb425ac17eefe6"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "bd1fe6940ea198db88527d359b1bc92e"
server_key_iv: "bd4aa638fdbac539762b89b887a18dde"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.523294 25556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal/instance:
uuid: "a2445c83ebe6491aacbb425ac17eefe6"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "bd1fe6940ea198db88527d359b1bc92e"
server_key_iv: "bd4aa638fdbac539762b89b887a18dde"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.524801 25556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 14:48:08.525665 25571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.525835 25556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:08.525941 25556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal
uuid: "a2445c83ebe6491aacbb425ac17eefe6"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "bd1fe6940ea198db88527d359b1bc92e"
server_key_iv: "bd4aa638fdbac539762b89b887a18dde"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.526067 25556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:08.534314 25556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:08.536851 25556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:08.537025 25556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:08.541055 25556 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.188:37387
I20260928 14:48:08.541083 25623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.188:37387 every 8 connection(s)
I20260928 14:48:08.541450 25556 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
I20260928 14:48:08.542547 25624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:08.544749 25624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:48:08.545935 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 25556
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.188@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.188@KRBTEST.COM
I20260928 14:48:08.555755 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.545653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.18.96.190:39113)
0928 14:48:08.545726 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.545728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.545784 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:48:08.546361 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.546362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.546393 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.546463 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.547245 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.548027 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.549263 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.549535 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.549614 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.552379 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.552380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.552381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.552434 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.554130 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.554558 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.554560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.554560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.554625 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.554926 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.554927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.554927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.555064 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.555137 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.555509 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.555636 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:48:08.555801 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.545465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42346)
0928 14:48:08.545641 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.545694 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.546221 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.546599 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.546604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.546984 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.547149 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.547153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.548132 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.548133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.549162 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.549164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.549641 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.549642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.549740 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.550293 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.550304 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.552271 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.554218 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.554223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.554225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.554483 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.554697 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.554697 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.554698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.554846 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.555142 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.555147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.555329 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.555438 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.555571 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260928 14:48:08.556479 25624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:48:08.556617 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.546485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.18.96.188:37387)
0928 14:48:08.546678 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.546682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.546698 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:48:08.546716 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.546718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.546738 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.546827 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.547474 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.548473 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.549441 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.549851 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.550020 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.552832 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.552841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.552841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.552849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.554710 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.555091 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.555094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.555095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.555154 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.555395 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.555396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.555397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.555632 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.555816 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.556401 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.556496 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.188@KRBTEST.COM for oryx/127.18.96.189@KRBTEST.COM
I20260928 14:48:08.563150 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.556777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.0.0.1:43470)
0928 14:48:08.556828 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.556837 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.556869 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.557008 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.557008 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.557116 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.557148 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.557149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.557887 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.557887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.558384 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.558385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.558748 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.558748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.558792 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.559229 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.559233 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.560512 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.561990 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.561991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.561991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.562332 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.562528 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.562528 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.562529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.562582 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.562815 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.562815 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.562853 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.562979 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.563080 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 14:48:08.563167 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.556830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.18.96.189:42805)
0928 14:48:08.556879 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.556881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.556890 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:08.556918 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.556919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.556934 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.557009 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.557222 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.557819 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.558459 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.558681 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.558741 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.560593 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.560594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.560594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.560601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.561912 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.562407 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.562407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.562408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.562465 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.562638 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.562639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.562639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.562759 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.562815 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.563007 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.563098 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:48:08.563720 25624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.188@KRBTEST.COM for oryx/127.18.96.188@KRBTEST.COM
I20260928 14:48:08.570930 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.564141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.0.0.1:58088)
0928 14:48:08.564189 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.564199 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.564242 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.564370 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.564370 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.564490 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.564523 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.564524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.565212 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.565212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.565677 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.565679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.566171 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.566172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.566211 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.566716 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.566719 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.568193 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.569852 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.569853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.569854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.570110 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.570361 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.570362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.570362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.570418 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.570677 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.570677 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.570714 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.570787 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.570859 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 14:48:08.571118 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.564207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.18.96.188:37387)
0928 14:48:08.564246 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.564248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.564285 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:48:08.564302 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.564302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.564313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.564355 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.564591 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.565103 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.565787 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.566092 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.566159 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.568275 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.568277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.568277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.568283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.569777 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.570240 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.570241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.570241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.570295 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.570502 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.570503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.570503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.570613 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.570671 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.570837 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.570998 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:48:08.572652 25624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: Bootstrap starting.
I20260928 14:48:08.573554 25624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:08.573855 25624 log.cc:824] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:08.574606 25624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: No bootstrap required, opened a new log
I20260928 14:48:08.576314 25624 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } }
I20260928 14:48:08.576536 25624 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:08.576592 25624 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2445c83ebe6491aacbb425ac17eefe6, State: Initialized, Role: FOLLOWER
I20260928 14:48:08.576762 25624 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } }
I20260928 14:48:08.577081 25629 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } } }
I20260928 14:48:08.577133 25624 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:08.577169 25629 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:48:08.578917 25640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:48:08.578970 25640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:48:08.581442 25556 master_runner.cc:428] Detected that this master a2445c83ebe6491aacbb425ac17eefe6 is joining an existing cluster
I20260928 14:48:08.581506 25556 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.96.188:37387
I20260928 14:48:08.590812 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.582885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.0.0.1:58096)
0928 14:48:08.583479 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.583488 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.583556 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.583697 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.583698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.583921 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.583981 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.583982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.584888 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.584888 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.585821 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.585823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.586337 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.586338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.586392 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.587413 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.587417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.587926 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.589824 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.589825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.589825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.590011 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.590217 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.590218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.590218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.590275 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.590544 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.590545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.590583 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.590638 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.590726 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":63,"thread_start_us":78,"threads_started":1}
I20260928 14:48:08.590909 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.582890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.18.96.188:37387)
0928 14:48:08.582926 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.582928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.583536 (+ 608us) server_negotiation.cc:407] Connection header received
0928 14:48:08.583615 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.583616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.583623 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.583649 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.584076 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.584800 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.585912 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.586206 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.586246 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.588012 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.588014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.588014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.588022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.589731 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.590088 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.590090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.590090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.590148 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.590337 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.590338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.590339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.590473 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.590542 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.590676 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.590788 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":18}
I20260928 14:48:08.608124 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 24967
I20260928 14:48:08.608806 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.582651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.18.96.190:39113)
0928 14:48:08.582695 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.582698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.583095 (+ 397us) server_negotiation.cc:407] Connection header received
0928 14:48:08.583200 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.583201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.583212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.583242 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.583817 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.584711 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.586226 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.586990 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.587029 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.605880 (+ 18851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.605882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.605882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.605891 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.607575 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.608031 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.608032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.608033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.608089 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.608284 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.608285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.608285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.608396 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.608431 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.608617 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.608704 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:48:08.609232 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.582579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42354)
0928 14:48:08.583029 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.583044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.583120 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.583306 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.583306 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.583625 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.583709 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.583712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.584815 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.584816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.586118 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.586119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.587106 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.587106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.587158 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.604684 (+ 17526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.604690 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.605434 (+ 744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.607680 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.607681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.607682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.607930 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.608161 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.608162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.608162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.608216 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.608468 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.608469 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.608508 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.608927 (+ 419us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.609119 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":104,"spinlock_wait_cycles":10624,"thread_start_us":89,"threads_started":1}
I20260928 14:48:08.609825 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.582775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.18.96.189:42805)
0928 14:48:08.582816 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.582818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.583095 (+ 277us) server_negotiation.cc:407] Connection header received
0928 14:48:08.583181 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.583183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.583194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.583243 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.583817 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.584705 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.586226 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.586554 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.586599 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.605566 (+ 18967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.605569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.605569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.605579 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.607287 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.607773 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.607774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.607775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.607835 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.608098 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.608099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.608099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.608257 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.608301 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.609588 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.609682 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:48:08.610369 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.582577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.0.0.1:43484)
0928 14:48:08.583008 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.583017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.583109 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.583257 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.583258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.583607 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.583710 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.583712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.584815 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.584816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.586113 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.586115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.586678 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.586679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.586746 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.604774 (+ 18028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.604779 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.605421 (+ 642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.607423 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.607424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.607425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.607680 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.607956 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.607957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.607957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.608025 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.609175 (+ 1150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.609176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.609228 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.610152 (+ 924us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.610261 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":55,"thread_start_us":86,"threads_started":1}
I20260928 14:48:08.614524 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:38549
--local_ip_for_outbound_sockets=127.18.96.129
--tserver_master_addrs=127.18.96.190:39113
--webserver_port=34777
--webserver_interface=127.18.96.129
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab
--principal=oryx/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.96.190:39113,127.18.96.189:42805
--tserver_master_addrs=127.18.96.190:39113,127.18.96.189:42805,127.18.96.188:37387 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606887, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.190@KRBTEST.COM for oryx/127.18.96.188@KRBTEST.COM
I20260928 14:48:08.622792 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.614507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.18.96.188:37387)
0928 14:48:08.614564 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.614566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.614572 (+ 6us) server_negotiation.cc:407] Connection header received
0928 14:48:08.614611 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.614611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.614623 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.614679 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.615075 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.615548 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.616636 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.616941 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.617029 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.619395 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.619397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.619397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.619404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.621139 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.621595 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.621597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.621597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.621668 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.621926 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.621928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.621928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.622118 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.622231 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.622609 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.622691 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:48:08.622890 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.614348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.0.0.1:58112)
0928 14:48:08.614411 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.614419 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.614451 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.614741 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.614742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.614927 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.614987 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.614989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.615638 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.615639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.616539 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.616542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.617032 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.617033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.617113 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.617577 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.617583 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.619299 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.621253 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.621254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.621255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.621481 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.621754 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.621755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.621755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.621821 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.622334 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.622335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.622401 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.622570 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.622725 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:48:08.623579 24905 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.96.188:37387: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6"
member_type: NON_VOTER
last_known_addr {
host: "127.18.96.188"
port: 37387
}
attrs {
promote: true
}
}
dest_uuid: "db5e569e3cd147088982678eeae71397"
cas_config_opid_index: 5
I20260928 14:48:08.623829 24905 consensus_queue.cc:237] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } }
I20260928 14:48:08.624511 25464 raft_consensus.cc:2932] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2445c83ebe6491aacbb425ac17eefe6 (127.18.96.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.625752 25464 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.96.188:37387
I20260928 14:48:08.625835 25464 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.625938 25464 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:08.626273 25556 master.cc:537] Master@127.18.96.188:37387 shutting down...
I20260928 14:48:08.626379 25598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:08.627348 25598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 FOLLOWER]: Refusing update from remote peer db5e569e3cd147088982678eeae71397: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:48:08.627743 25463 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:48:08.629038 24892 consensus_peers.cc:487] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 -> Peer a2445c83ebe6491aacbb425ac17eefe6 (127.18.96.188:37387): Couldn't send request to peer a2445c83ebe6491aacbb425ac17eefe6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:48:08.630357 25556 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:08.630506 25556 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:08.630610 25556 tablet_replica.cc:330] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: stopping tablet replica
I20260928 14:48:08.633489 25556 master.cc:552] Master@127.18.96.188:37387 shutdown complete.
I20260928 14:48:08.633574 25556 master_runner.cc:305] Clearing existing system tablet
I20260928 14:48:08.633754 25556 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:08.633817 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.624330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.0.0.1:43500)
0928 14:48:08.629496 (+ 5166us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.629510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.629567 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.629807 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.629808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.629810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:08.629972 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.629974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.631028 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.631029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.632463 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.632466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.633544 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.633545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.633619 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.633643 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.633698 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.633739 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5074}
I20260928 14:48:08.633841 25556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:08.634097 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.624457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.18.96.189:42805)
0928 14:48:08.624513 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.624514 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.629628 (+ 5114us) server_negotiation.cc:407] Connection header received
0928 14:48:08.629660 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.629661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.629683 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.629743 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:08.630055 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.630928 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.632569 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.633347 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.633434 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.633507 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.633985 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.634027 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:48:08.634305 25512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 LEARNER]: Refusing update from remote peer db5e569e3cd147088982678eeae71397: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:48:08.634526 25463 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 14:48:08.634917 25556 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:48:08.635092 25512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2445c83ebe6491aacbb425ac17eefe6 (127.18.96.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.635479 25652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.635605 25556 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:08.635649 25556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.635670 25556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal
uuid: "a2445c83ebe6491aacbb425ac17eefe6"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "bd1fe6940ea198db88527d359b1bc92e"
server_key_iv: "bd4aa638fdbac539762b89b887a18dde"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.636292 25542 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db5e569e3cd147088982678eeae71397. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.636391 25542 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:48:08.636436 25542 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: NON_VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: true } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.636489 25542 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:48:08.636816 25556 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:08.638295 25556 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:48:08.638358 25556 log.cc:1196] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 14:48:08.638587 25556 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: Deleting consensus metadata
I20260928 14:48:08.638700 25556 master_runner.cc:315] Copying system tablet from 127.18.96.190:39113
I20260928 14:48:08.639315 25556 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.96.190:39113
I20260928 14:48:08.640470 25463 raft_consensus.cc:1050] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: attempting to promote NON_VOTER ec8f238e20044baf9d7ce74d06e053a2 to VOTER
I20260928 14:48:08.640733 25463 consensus_queue.cc:237] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } }
I20260928 14:48:08.641428 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.641189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.0.0.1:58124)
0928 14:48:08.641334 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.641384 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 14:48:08.641680 25512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 LEARNER]: Refusing update from remote peer db5e569e3cd147088982678eeae71397: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 14:48:08.641986 25464 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 14:48:08.642339 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.642205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.0.0.1:58132)
0928 14:48:08.642285 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.642313 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:37387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 14:48:08.642818 24892 consensus_peers.cc:487] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 -> Peer a2445c83ebe6491aacbb425ac17eefe6 (127.18.96.188:37387): Couldn't send request to peer a2445c83ebe6491aacbb425ac17eefe6. Status: Network error: Client connection negotiation failed: client connection to 127.18.96.188:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:48:08.643072 25464 raft_consensus.cc:2932] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ec8f238e20044baf9d7ce74d06e053a2 (127.18.96.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.644119 25512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ec8f238e20044baf9d7ce74d06e053a2 (127.18.96.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.645699 25464 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.645782 25464 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:08.645830 25464 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.645869 25464 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:08.646386 25653 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.646499 25653 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:08.647627 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.639671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.18.96.190:39113)
0928 14:48:08.639730 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.639732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.639926 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:48:08.640094 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.640096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.640107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.640140 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.640596 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.641371 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.642751 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.643062 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.643103 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.644557 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.644558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.644558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.644565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.646235 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.646800 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.646802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.646802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.646871 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.647083 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.647084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.647084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.647207 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.647256 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.647464 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.647542 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:48:08.648164 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.639732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42358)
0928 14:48:08.639825 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.639836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.639953 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.640206 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.640207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.640414 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.640458 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.640460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.641590 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.641591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.642652 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.642655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.643268 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.643268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.643326 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.643953 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.643959 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.644457 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.646398 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.646399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.646400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.646712 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.646941 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.646942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.646942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.646998 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.647300 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.647301 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.647350 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.647912 (+ 562us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.648032 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":65,"threads_started":1}
I20260928 14:48:08.648614 24925 tablet_copy_service.cc:140] P db5e569e3cd147088982678eeae71397: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a2445c83ebe6491aacbb425ac17eefe6 ({username='oryx', principal='oryx/127.18.96.188@KRBTEST.COM'} at 127.0.0.1:42358)
I20260928 14:48:08.648664 24925 tablet_copy_service.cc:161] P db5e569e3cd147088982678eeae71397: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a2445c83ebe6491aacbb425ac17eefe6 at {username='oryx', principal='oryx/127.18.96.188@KRBTEST.COM'} at 127.0.0.1:42358: session id = a2445c83ebe6491aacbb425ac17eefe6-00000000000000000000000000000000
I20260928 14:48:08.649224 24925 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:48:08.649840 25556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:48:08.651114 25556 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: tablet copy: Starting download of 0 data blocks...
I20260928 14:48:08.651242 25556 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: tablet copy: Starting download of 1 WAL segments...
I20260928 14:48:08.652101 25556 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:48:08.652953 24925 tablet_copy_service.cc:342] P db5e569e3cd147088982678eeae71397: Request end of tablet copy session a2445c83ebe6491aacbb425ac17eefe6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.96.188@KRBTEST.COM'} at 127.0.0.1:42358
I20260928 14:48:08.653000 24925 tablet_copy_service.cc:433] P db5e569e3cd147088982678eeae71397: ending tablet copy session a2445c83ebe6491aacbb425ac17eefe6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a2445c83ebe6491aacbb425ac17eefe6
I20260928 14:48:08.655468 25556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:08.656725 25661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.656775 25664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.657168 25662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.661268 25556 server_base.cc:1034] running on GCE node
I20260928 14:48:08.661389 25556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:08.661554 25556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:08.662690 25556 hybrid_clock.cc:648] HybridClock initialized: now 1790606888662668 us; error 41 us; skew 500 ppm
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:08.664314 25556 init.cc:377] Logged in from keytab as oryx/127.18.96.188@KRBTEST.COM (short username oryx)
I20260928 14:48:08.664583 25556 webserver.cc:460] Webserver started at http://127.18.96.188:36999/ using document root <none> and password file <none>
I20260928 14:48:08.664708 25556 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:08.664748 25556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:08.665560 25556 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:08.666174 25671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.666337 25556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.666389 25556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal
uuid: "a2445c83ebe6491aacbb425ac17eefe6"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "bd1fe6940ea198db88527d359b1bc92e"
server_key_iv: "bd4aa638fdbac539762b89b887a18dde"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.666448 25556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:08.701138 25556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:08.703361 25556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:08.706465 25556 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.188:37387
I20260928 14:48:08.706894 25556 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
I20260928 14:48:08.707968 25724 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:48:08.708509 25724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: Bootstrap starting.
I20260928 14:48:08.711651 25724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:48:08.711902 25724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: Bootstrap complete.
I20260928 14:48:08.712102 25724 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } }
I20260928 14:48:08.712194 25724 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2445c83ebe6491aacbb425ac17eefe6, State: Initialized, Role: LEARNER
I20260928 14:48:08.712227 25724 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } }
I20260928 14:48:08.712404 25724 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:08.714175 25738 catalog_manager.cc:1259] Loaded cluster ID: 0c27a911cfc5475a9bc64e29ac98a8b4
I20260928 14:48:08.714334 25738 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: loading cluster ID for follower catalog manager: success
I20260928 14:48:08.715508 25738 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: acquiring CA information for follower catalog manager: success
I20260928 14:48:08.715906 25738 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:48:08.715996 25723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.188:37387 every 8 connection(s)
I20260928 14:48:08.720019 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:08.720156 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: This master's current role is: LEARNER
W20260928 14:48:08.725199 25649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:08.725409 25649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:08.725433 25649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:08.726619 25649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:08.726660 25649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:08.726686 25649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:08.728266 25649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab
--principal=oryx/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:38549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=34777
--tserver_master_addrs=127.18.96.190:39113,127.18.96.189:42805,127.18.96.188:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:08.728506 25649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:08.728765 25649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:08.733887 25743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.733916 25744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.733888 25746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.734341 25649 server_base.cc:1034] running on GCE node
I20260928 14:48:08.734543 25649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:08.734817 25649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:08.736016 25649 hybrid_clock.cc:648] HybridClock initialized: now 1790606888735952 us; error 29 us; skew 500 ppm
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:08.738745 25649 init.cc:377] Logged in from keytab as oryx/127.18.96.129@KRBTEST.COM (short username oryx)
I20260928 14:48:08.739306 25649 webserver.cc:460] Webserver started at http://127.18.96.129:34777/ using document root <none> and password file <none>
I20260928 14:48:08.739538 25649 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:08.739624 25649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:08.741124 25649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:08.741794 25753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.742018 25649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:08.742118 25649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "02c3b535fb9541e89be93fcb8d8155c1"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "eba46694bdc94a1eaa3296fe8119c517"
server_key_iv: "5119ceacde5df536516aa9323c5bf8eb"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.742496 25649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:08.759454 25649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:08.761956 25649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:08.762147 25649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:08.762393 25649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:08.762724 25649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:08.762778 25649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.762823 25649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:08.762847 25649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.769300 25649 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:38549
I20260928 14:48:08.769343 25866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:38549 every 8 connection(s)
I20260928 14:48:08.769711 25649 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:08.771350 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 25649
I20260928 14:48:08.771533 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 25101
I20260928 14:48:08.776911 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:44377
--local_ip_for_outbound_sockets=127.18.96.130
--tserver_master_addrs=127.18.96.190:39113
--webserver_port=45529
--webserver_interface=127.18.96.130
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab
--principal=oryx/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.96.190:39113,127.18.96.189:42805
--tserver_master_addrs=127.18.96.190:39113,127.18.96.189:42805,127.18.96.188:37387 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.189@KRBTEST.COM
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.188@KRBTEST.COM
I20260928 14:48:08.787572 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.771110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:41179 (local address 127.18.96.190:39113)
0928 14:48:08.771175 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.771178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.771206 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:48:08.777078 (+ 5872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.777080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.777094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.777127 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.778374 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.779184 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.780420 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.780725 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.780791 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.784003 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.784004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.784004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.784011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.785659 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.786396 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.786397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.786397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.786457 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.786811 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.786812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.786812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.786960 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.787014 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.787383 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.787474 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:48:08.787636 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.770523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:47759 (local address 127.18.96.189:42805)
0928 14:48:08.770565 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.770567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.770826 (+ 259us) server_negotiation.cc:407] Connection header received
0928 14:48:08.777107 (+ 6281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.777110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.777127 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.777159 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.778388 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.779157 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.780504 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.780801 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.780860 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.783530 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.783532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.783532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.783541 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.785341 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.786834 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.786836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.786836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.786887 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.787083 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.787084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.787085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.787214 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.787256 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.787453 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.787520 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:48:08.788236 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.771024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.129:41179)
0928 14:48:08.771154 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.771164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.776937 (+ 5773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.777725 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.777726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.777981 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.778248 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.778255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.779284 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.779285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.780324 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.780327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.780814 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.780816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.780963 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.781763 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.781768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.783904 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.785747 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.785755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.785757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.786293 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.786528 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.786528 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.786528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.786724 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.787045 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.787052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.787223 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.787710 (+ 487us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.787982 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":186,"thread_start_us":78,"threads_started":1}
I20260928 14:48:08.788236 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.770495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.18.96.129:47759)
0928 14:48:08.770670 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.770735 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.776929 (+ 6194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.777339 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.777344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.777943 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.778279 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.778282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.779278 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.779279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.780431 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.780434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.780907 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.780908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.780971 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.781658 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.781668 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.783423 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.786528 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.786528 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.786528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.786746 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.786959 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.786959 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.786960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.787011 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.787302 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.787303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.787341 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.787724 (+ 383us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.787939 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":81,"thread_start_us":100,"threads_started":1}
I20260928 14:48:08.788941 25869 heartbeater.cc:347] Connected to a master server at 127.18.96.189:42805
I20260928 14:48:08.789039 25869 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:08.789238 25869 heartbeater.cc:511] Master 127.18.96.189:42805 requested a full tablet report, sending...
I20260928 14:48:08.789405 25868 heartbeater.cc:347] Connected to a master server at 127.18.96.190:39113
I20260928 14:48:08.789448 25868 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:08.789590 25868 heartbeater.cc:511] Master 127.18.96.190:39113 requested a full tablet report, sending...
I20260928 14:48:08.789749 25502 ts_manager.cc:194] Registered new tserver with Master: 02c3b535fb9541e89be93fcb8d8155c1 (127.18.96.129:38549)
I20260928 14:48:08.789824 24905 ts_manager.cc:194] Re-registered known tserver with Master: 02c3b535fb9541e89be93fcb8d8155c1 (127.18.96.129:38549)
I20260928 14:48:08.790267 24905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:41179
I20260928 14:48:08.790935 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.770897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:54429 (local address 127.18.96.188:37387)
0928 14:48:08.771286 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.771288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.771299 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:08.777371 (+ 6072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.777372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.777388 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.777449 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.779270 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.780322 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.781665 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.781991 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.782058 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.786535 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.786537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.786537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.786546 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.789542 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.790049 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.790050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.790050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.790114 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.790368 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.790369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.790369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.790515 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.790580 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.790735 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.790831 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":55,"threads_started":1}
I20260928 14:48:08.791229 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.770825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.18.96.129:54429)
0928 14:48:08.771148 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.771161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.776927 (+ 5766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.777509 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.777509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.777934 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.779155 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.779157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.780849 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.780850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.781553 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.781555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.782120 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.782121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.782181 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.783989 (+ 1808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.783993 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.786440 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.789668 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.789669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.789670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.789951 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.790191 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.790192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.790193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.790259 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.790572 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.790573 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.790623 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.790696 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.790811 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260928 14:48:08.791604 25867 heartbeater.cc:347] Connected to a master server at 127.18.96.188:37387
I20260928 14:48:08.791663 25867 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:08.791728 25867 heartbeater.cc:511] Master 127.18.96.188:37387 requested a full tablet report, sending...
I20260928 14:48:08.792176 25688 ts_manager.cc:194] Registered new tserver with Master: 02c3b535fb9541e89be93fcb8d8155c1 (127.18.96.129:38549)
W20260928 14:48:08.864715 25875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:08.864992 25875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:08.865018 25875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:08.866235 25875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:08.866291 25875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:08.866356 25875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:08.868089 25875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab
--principal=oryx/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:44377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=45529
--tserver_master_addrs=127.18.96.190:39113,127.18.96.189:42805,127.18.96.188:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:08.868283 25875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:08.868460 25875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:08.873787 25881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:08.873875 25883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.873982 25875 server_base.cc:1034] running on GCE node
W20260928 14:48:08.873787 25880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:08.874284 25875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:08.874560 25875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:08.875787 25875 hybrid_clock.cc:648] HybridClock initialized: now 1790606888875749 us; error 36 us; skew 500 ppm
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:08.878990 25875 init.cc:377] Logged in from keytab as oryx/127.18.96.130@KRBTEST.COM (short username oryx)
I20260928 14:48:08.879490 25875 webserver.cc:460] Webserver started at http://127.18.96.130:45529/ using document root <none> and password file <none>
I20260928 14:48:08.879756 25875 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:08.879853 25875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:08.881448 25875 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:08.882190 25890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.882444 25875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:08.882560 25875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "2f6fc171f9f943a7bb6100700bc6c615"
format_stamp: "Formatted at 2026-09-28 14:48:07 on dist-test-slave-j968"
server_key: "4e8f07ca9c6e6913e96eff9ac5b3c800"
server_key_iv: "9995dcb829a73409b7bb9d2740dc7edc"
server_key_version: "encryptionkey@0"
I20260928 14:48:08.882926 25875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:08.905373 25875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:08.907531 25875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:08.907714 25875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:08.908036 25875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:08.908368 25875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:08.908432 25875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.908473 25875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:08.908501 25875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:08.914829 25875 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:44377
I20260928 14:48:08.914875 26003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:44377 every 8 connection(s)
I20260928 14:48:08.915340 25875 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.189@KRBTEST.COM
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
Sep 28 14:48:08 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606888, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.188@KRBTEST.COM
I20260928 14:48:08.924304 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 25875
I20260928 14:48:08.924530 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 25235
I20260928 14:48:08.929457 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.916480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:52629 (local address 127.18.96.190:39113)
0928 14:48:08.916537 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.916540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.916656 (+ 116us) server_negotiation.cc:407] Connection header received
0928 14:48:08.917657 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.917658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.917669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.917817 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.918681 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.919449 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.920970 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.921233 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.921302 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.925971 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.925972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.925973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.925981 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.927930 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.928331 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.928332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.928332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.928396 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.928621 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.928623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.928623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.928752 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.928820 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.929276 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.929387 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:48:08.929641 26010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.916125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.130:52629)
0928 14:48:08.916496 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.916552 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.917527 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.917908 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.917913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.918393 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.918566 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.918572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.919540 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.919541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.920507 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.920509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.921317 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.921318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.921372 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.921964 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.921970 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.924616 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.928011 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.928012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.928013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.928247 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.928457 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.928458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.928459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.928534 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.928819 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.928826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.929010 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.929140 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.929332 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":46,"thread_start_us":97,"threads_started":1}
I20260928 14:48:08.930121 26005 heartbeater.cc:347] Connected to a master server at 127.18.96.190:39113
I20260928 14:48:08.930222 26005 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:08.930320 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.916334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:34643 (local address 127.18.96.188:37387)
0928 14:48:08.916415 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.916420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.917903 (+ 1483us) server_negotiation.cc:407] Connection header received
0928 14:48:08.917986 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.917987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.918002 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.918061 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.918681 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.919440 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.920587 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.920864 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.920924 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.925956 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.925958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.925958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.925965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.927564 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.928245 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.928246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.928246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.928310 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.928598 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.928599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.928600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.928752 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.928820 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.930070 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.930200 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:48:08.930418 26005 heartbeater.cc:511] Master 127.18.96.190:39113 requested a full tablet report, sending...
I20260928 14:48:08.931001 24905 ts_manager.cc:194] Re-registered known tserver with Master: 2f6fc171f9f943a7bb6100700bc6c615 (127.18.96.130:44377)
I20260928 14:48:08.931022 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.916244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.18.96.130:34643)
0928 14:48:08.917844 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.917852 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.917927 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.918082 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.918083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.918352 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.918567 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.918572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.919540 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.919541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.920512 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.920513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.920983 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.920984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.921067 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.921608 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.921616 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.925845 (+ 4229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.927674 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.927682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.927684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.928154 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.928369 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.928370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.928370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.928519 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.929869 (+ 1350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.929870 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.929916 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.929974 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.930866 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1578,"mutex_wait_us":88,"thread_start_us":55,"threads_started":1}
I20260928 14:48:08.931504 24905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:52629
I20260928 14:48:08.931933 26004 heartbeater.cc:347] Connected to a master server at 127.18.96.188:37387
I20260928 14:48:08.932016 26004 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:08.932287 26004 heartbeater.cc:511] Master 127.18.96.188:37387 requested a full tablet report, sending...
I20260928 14:48:08.932287 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:40059
--local_ip_for_outbound_sockets=127.18.96.131
--tserver_master_addrs=127.18.96.190:39113
--webserver_port=38311
--webserver_interface=127.18.96.131
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab
--principal=oryx/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.96.190:39113,127.18.96.189:42805
--tserver_master_addrs=127.18.96.190:39113,127.18.96.189:42805,127.18.96.188:37387 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:08.932752 25688 ts_manager.cc:194] Registered new tserver with Master: 2f6fc171f9f943a7bb6100700bc6c615 (127.18.96.130:44377)
I20260928 14:48:08.933444 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.916449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:58777 (local address 127.18.96.189:42805)
0928 14:48:08.916504 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:08.916506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:08.917754 (+ 1248us) server_negotiation.cc:407] Connection header received
0928 14:48:08.917828 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:08.917829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:08.917841 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:08.917896 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:08.918663 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.919163 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.920398 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.920642 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.920708 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.923842 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:08.923845 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:08.923845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:08.923854 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:08.931536 (+ 7682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.932588 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.932589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.932589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.932648 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:08.932844 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:08.932845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:08.932846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:08.932984 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:08.933024 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:08.933298 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 14:48:08.933372 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:48:08.933470 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:08.916125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.18.96.130:58777)
0928 14:48:08.917677 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:08.917686 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:08.917757 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:08.917988 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:08.917989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:08.918337 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:08.918565 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.918570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.919271 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.919272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.920293 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:08.920295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:08.920750 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:08.920752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:08.920965 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:08.921692 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:08.921698 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:08.923698 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:08.931639 (+ 7941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.931640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.931641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.932477 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.932715 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:08.932716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:08.932716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:08.932779 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:08.933070 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:08.933070 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:08.933109 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 14:48:08.933179 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:48:08.933303 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"mutex_wait_us":29,"thread_start_us":124,"threads_started":1}
I20260928 14:48:08.934033 26006 heartbeater.cc:347] Connected to a master server at 127.18.96.189:42805
I20260928 14:48:08.934089 26006 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:08.934183 26006 heartbeater.cc:511] Master 127.18.96.189:42805 requested a full tablet report, sending...
I20260928 14:48:08.934526 25502 ts_manager.cc:194] Registered new tserver with Master: 2f6fc171f9f943a7bb6100700bc6c615 (127.18.96.130:44377)
W20260928 14:48:09.015448 26011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:09.015674 26011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:09.015698 26011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:09.016974 26011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:09.017026 26011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:09.017082 26011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:09.018538 26011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab
--principal=oryx/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:40059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=38311
--tserver_master_addrs=127.18.96.190:39113,127.18.96.189:42805,127.18.96.188:37387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:09.018777 26011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:09.019042 26011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:48:09.024297 26011 server_base.cc:1034] running on GCE node
W20260928 14:48:09.024281 26019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:09.024281 26017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:09.024266 26016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:09.024672 26011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:09.024909 26011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:09.026062 26011 hybrid_clock.cc:648] HybridClock initialized: now 1790606889026050 us; error 31 us; skew 500 ppm
Sep 28 14:48:09 dist-test-slave-j968 krb5kdc[24860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606889, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:09.028548 26011 init.cc:377] Logged in from keytab as oryx/127.18.96.131@KRBTEST.COM (short username oryx)
I20260928 14:48:09.029001 26011 webserver.cc:460] Webserver started at http://127.18.96.131:38311/ using document root <none> and password file <none>
I20260928 14:48:09.029227 26011 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:09.029309 26011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:09.030630 26011 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 14:48:09.031243 26026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:09.031450 26011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:48:09.031556 26011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "c4bbac146304448287d481482d4cc6bb"
format_stamp: "Formatted at 2026-09-28 14:48:08 on dist-test-slave-j968"
server_key: "8cdb76ad99f038661c7e6faec2a8ec3b"
server_key_iv: "55f01e2de3d51cb37591cb07e9d47a72"
server_key_version: "encryptionkey@0"
I20260928 14:48:09.031934 26011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:09.064327 26011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:09.066354 26011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:09.066465 26011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:09.066664 26011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:09.067003 26011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:09.067039 26011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:09.067092 26011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:09.067116 26011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:09.073621 26011 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:40059
I20260928 14:48:09.073696 26139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:40059 every 8 connection(s)
I20260928 14:48:09.074033 26011 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:09.078194 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 26011
Sep 28 14:48:09 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606889, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.189@KRBTEST.COM
Sep 28 14:48:09 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606889, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.188@KRBTEST.COM
Sep 28 14:48:09 dist-test-slave-j968 krb5kdc[24860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606889, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:09.088120 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.074725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.131:53125)
0928 14:48:09.074989 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:09.075046 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:09.075628 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:09.075906 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:09.075911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:09.076382 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:09.076673 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.076678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.077526 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.077527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.078965 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.078968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.079491 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.079492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.079670 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.080487 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:09.080498 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:09.084038 (+ 3540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:09.086015 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:09.086024 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:09.086027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:09.086600 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:09.086807 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:09.086808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:09.086808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:09.086950 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:09.087271 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:09.087277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:09.087477 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 14:48:09.087616 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 14:48:09.087805 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":109,"thread_start_us":86,"threads_started":1}
I20260928 14:48:09.088526 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.074785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:53125 (local address 127.18.96.190:39113)
0928 14:48:09.074850 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.074852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.075149 (+ 297us) server_negotiation.cc:407] Connection header received
0928 14:48:09.075730 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.075731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.075744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.075800 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:09.076763 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.077358 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.079073 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.079356 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.079397 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.084129 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:09.084130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:09.084131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:09.084138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:09.085919 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.086677 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.086678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.086678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.086756 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.087013 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.087014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.087014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.087155 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:09.087224 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.088340 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.088417 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:48:09.089028 26141 heartbeater.cc:347] Connected to a master server at 127.18.96.190:39113
I20260928 14:48:09.089133 26141 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:09.089316 26141 heartbeater.cc:511] Master 127.18.96.190:39113 requested a full tablet report, sending...
I20260928 14:48:09.089406 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.074725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.18.96.131:42699)
0928 14:48:09.075219 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:09.075230 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:09.075665 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:09.076323 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:09.076323 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:09.076541 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:09.076674 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.076680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.077640 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.077640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.078967 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.078970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.079743 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.079743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.079819 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.080498 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:09.080504 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:09.083687 (+ 3183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:09.087590 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:09.087591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:09.087591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:09.088351 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:09.088583 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:09.088584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:09.088584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:09.088649 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:09.088937 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:09.088937 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:09.088979 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:48:09.089048 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:48:09.089152 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":446,"thread_start_us":87,"threads_started":1}
I20260928 14:48:09.089522 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.075031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:32909 (local address 127.18.96.189:42805)
0928 14:48:09.075102 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.075104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.075284 (+ 180us) server_negotiation.cc:407] Connection header received
0928 14:48:09.075781 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.075782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.075795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.075828 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:09.076788 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.077538 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.079138 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.079422 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.079460 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.084734 (+ 5274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:09.084736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:09.084736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:09.084744 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:09.087964 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.088666 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.088667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.088668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.088720 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.088923 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.088923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.088924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.089050 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:09.089094 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.089308 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.089406 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:48:09.089671 24905 ts_manager.cc:194] Re-registered known tserver with Master: c4bbac146304448287d481482d4cc6bb (127.18.96.131:40059)
I20260928 14:48:09.089745 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.074969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:42699 (local address 127.18.96.188:37387)
0928 14:48:09.075044 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.075046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.075277 (+ 231us) server_negotiation.cc:407] Connection header received
0928 14:48:09.075781 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.075782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.075795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.075828 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:09.076787 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.077548 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.079077 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.079397 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.079441 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.083766 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:09.083768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:09.083768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:09.083776 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:09.087482 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.088445 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.088446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.088446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.088502 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.088716 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.088717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.088717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.088827 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:09.088908 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.089577 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.089666 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:48:09.089898 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.074725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.18.96.131:32909)
0928 14:48:09.075198 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:09.075210 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:09.075669 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:09.076006 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:09.076007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:09.076391 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:09.076674 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.076680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.077640 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.077640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.078965 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.078968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.079528 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.079528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.079670 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.080487 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:09.080498 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:09.084637 (+ 4139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:09.088235 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:09.088236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:09.088236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:09.088594 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:09.088784 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:09.088785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:09.088785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:09.088845 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:09.089144 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:09.089144 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:09.089186 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:48:09.089635 (+ 449us) client_negotiation.cc:240] Negotiation successful
0928 14:48:09.089749 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":107,"thread_start_us":84,"threads_started":1}
I20260928 14:48:09.090014 26140 heartbeater.cc:347] Connected to a master server at 127.18.96.188:37387
I20260928 14:48:09.090067 26140 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:09.090121 26140 heartbeater.cc:511] Master 127.18.96.188:37387 requested a full tablet report, sending...
I20260928 14:48:09.090121 24905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:53125
I20260928 14:48:09.090229 26142 heartbeater.cc:347] Connected to a master server at 127.18.96.189:42805
I20260928 14:48:09.090270 26142 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:09.090327 26142 heartbeater.cc:511] Master 127.18.96.189:42805 requested a full tablet report, sending...
I20260928 14:48:09.090378 25688 ts_manager.cc:194] Registered new tserver with Master: c4bbac146304448287d481482d4cc6bb (127.18.96.131:40059)
I20260928 14:48:09.090582 25502 ts_manager.cc:194] Registered new tserver with Master: c4bbac146304448287d481482d4cc6bb (127.18.96.131:40059)
I20260928 14:48:09.092836 26153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.079634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.18.96.189:42805)
0928 14:48:09.080008 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.080010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.081805 (+ 1795us) server_negotiation.cc:407] Connection header received
0928 14:48:09.081976 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.081977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.081990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.082034 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:09.082899 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.083634 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.085740 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.086072 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.086122 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.088008 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:09.088009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:09.088009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:09.088016 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:09.091532 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.091911 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.091912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.091913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.091974 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.092186 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.092187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.092187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.092317 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:09.092403 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.092639 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.092729 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":61,"threads_started":1}
I20260928 14:48:09.093797 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.079961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.18.96.190:39113)
0928 14:48:09.080120 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.080121 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.081934 (+ 1813us) server_negotiation.cc:407] Connection header received
0928 14:48:09.081958 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.081959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.081974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.082037 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:09.082830 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.083592 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.084681 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.084996 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.085062 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.086711 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:09.086712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:09.086713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:09.086720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:09.092301 (+ 5581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.092961 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.092962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.092962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.093023 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.093227 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.093228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.093229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.093359 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:09.093469 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.093660 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.093736 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":111,"threads_started":1}
I20260928 14:48:09.095117 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.079505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.18.96.188:37387)
0928 14:48:09.080061 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.080063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.080830 (+ 767us) server_negotiation.cc:407] Connection header received
0928 14:48:09.080946 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.080947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.080961 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.081032 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:09.081835 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.082655 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.084151 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.088137 (+ 3986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.088181 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.092106 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:09.092107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:09.092107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:09.092114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:09.093942 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.094281 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.094281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.094282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.094334 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.094532 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.094533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.094533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.094668 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:09.094762 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.094924 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.095012 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":58,"threads_started":1}
I20260928 14:48:09.095729 24905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:09.097383 24905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:09.105679 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.101669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.18.96.131:40059)
0928 14:48:09.101836 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.101841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.101850 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:09.101905 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.101907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.101921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.102122 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:09.102177 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.102991 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.104454 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.105323 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.105399 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.105483 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.105555 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.105597 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260928 14:48:09.105669 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.101375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:40059 (local address 127.0.0.1:39070)
0928 14:48:09.101500 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:09.101511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:09.101555 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:09.102020 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:09.102021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:09.102022 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:09.102051 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.102052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.103079 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.103080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.104137 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.104139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.105410 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.105411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.105471 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.105474 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:09.105540 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:48:09.105586 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260928 14:48:09.106166 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.101347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38549 (local address 127.0.0.1:56234)
0928 14:48:09.101507 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:09.101519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:09.101571 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:09.101918 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:09.101919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:09.101921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:09.101977 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.101979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.103465 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.103465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.104705 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.104707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.105928 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.105929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.105990 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.105994 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:48:09.106040 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 14:48:09.106078 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 14:48:09.106458 26163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.101519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:44377 (local address 127.0.0.1:35282)
0928 14:48:09.102064 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:09.102074 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:09.102148 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:09.102374 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:09.102375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:09.102377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:09.102432 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.102434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.104055 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.104057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.105132 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.105135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.106207 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.106208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.106279 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.106283 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:48:09.106331 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 14:48:09.106375 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":104,"threads_started":1}
I20260928 14:48:09.106464 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.101479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.18.96.129:38549)
0928 14:48:09.101668 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.101673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.101684 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:09.101729 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.101731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.101748 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.101829 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:09.102118 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.103362 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.104838 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.105577 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.105638 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.105712 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.106317 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.106374 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260928 14:48:09.106487 26074 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6b45f18f4146f7807b29afce5cea0a (DEFAULT_TABLE table=test-table [id=81ef397b1f4b4743a2cff4aa51a4be93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:09.106742 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.101764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.18.96.130:44377)
0928 14:48:09.101977 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.101982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.102216 (+ 234us) server_negotiation.cc:407] Connection header received
0928 14:48:09.102264 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.102266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.102282 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.102370 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:09.102504 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.103842 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.105228 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.106038 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.106105 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.106173 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.106614 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.106664 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260928 14:48:09.106904 26074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6b45f18f4146f7807b29afce5cea0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:09.107081 25801 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6b45f18f4146f7807b29afce5cea0a (DEFAULT_TABLE table=test-table [id=81ef397b1f4b4743a2cff4aa51a4be93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:09.107402 25801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6b45f18f4146f7807b29afce5cea0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:09.107304 25938 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6b45f18f4146f7807b29afce5cea0a (DEFAULT_TABLE table=test-table [id=81ef397b1f4b4743a2cff4aa51a4be93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:09.107616 25938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6b45f18f4146f7807b29afce5cea0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:09.110102 26166 tablet_bootstrap.cc:492] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: Bootstrap starting.
I20260928 14:48:09.110565 26167 tablet_bootstrap.cc:492] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: Bootstrap starting.
I20260928 14:48:09.110927 26166 tablet_bootstrap.cc:654] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:09.111131 26168 tablet_bootstrap.cc:492] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: Bootstrap starting.
I20260928 14:48:09.111176 26167 tablet_bootstrap.cc:654] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:09.111378 26166 log.cc:824] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:09.111466 26167 log.cc:824] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:09.111799 26168 tablet_bootstrap.cc:654] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:09.112164 26168 log.cc:824] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:09.112329 26167 tablet_bootstrap.cc:492] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: No bootstrap required, opened a new log
I20260928 14:48:09.112334 26166 tablet_bootstrap.cc:492] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: No bootstrap required, opened a new log
I20260928 14:48:09.112388 26166 ts_tablet_manager.cc:1397] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:09.112401 26167 ts_tablet_manager.cc:1397] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:48:09.112953 26168 tablet_bootstrap.cc:492] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: No bootstrap required, opened a new log
I20260928 14:48:09.113029 26168 ts_tablet_manager.cc:1397] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:09.114464 26168 raft_consensus.cc:348] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } }
I20260928 14:48:09.114617 26168 raft_consensus.cc:374] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:09.114660 26168 raft_consensus.cc:729] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6fc171f9f943a7bb6100700bc6c615, State: Initialized, Role: FOLLOWER
I20260928 14:48:09.114766 26168 consensus_queue.cc:260] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } }
I20260928 14:48:09.115028 26168 ts_tablet_manager.cc:1428] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:09.115130 26005 heartbeater.cc:503] Master 127.18.96.190:39113 was elected leader, sending a full tablet report...
I20260928 14:48:09.114974 26167 raft_consensus.cc:348] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } }
I20260928 14:48:09.115473 26167 raft_consensus.cc:374] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:09.115516 26167 raft_consensus.cc:729] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4bbac146304448287d481482d4cc6bb, State: Initialized, Role: FOLLOWER
I20260928 14:48:09.115638 26167 consensus_queue.cc:260] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } }
I20260928 14:48:09.115917 26167 ts_tablet_manager.cc:1428] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 14:48:09.114974 26166 raft_consensus.cc:348] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } }
I20260928 14:48:09.116259 26141 heartbeater.cc:503] Master 127.18.96.190:39113 was elected leader, sending a full tablet report...
I20260928 14:48:09.116521 26166 raft_consensus.cc:374] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:09.116590 26166 raft_consensus.cc:729] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c3b535fb9541e89be93fcb8d8155c1, State: Initialized, Role: FOLLOWER
I20260928 14:48:09.116719 26166 consensus_queue.cc:260] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } }
I20260928 14:48:09.116993 26166 ts_tablet_manager.cc:1428] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:48:09.117110 25868 heartbeater.cc:503] Master 127.18.96.190:39113 was elected leader, sending a full tablet report...
W20260928 14:48:09.166878 26007 tablet.cc:2369] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:09.200269 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.197492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.18.96.188:37387)
0928 14:48:09.197562 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.197564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.197574 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:09.197629 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.197629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.197650 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.197712 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:09.197851 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.198381 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.199424 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.199829 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.199938 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.200046 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.200108 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.200163 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:48:09.200376 26163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.197359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.0.0.1:58150)
0928 14:48:09.197471 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:09.197489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:09.197570 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:09.197735 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:09.197735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:09.197737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:09.197791 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.197793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.198479 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.198481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.199313 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.199316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.199948 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.199949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.200031 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.200035 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:48:09.200138 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:48:09.200192 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 14:48:09.200793 25698 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 14:48:09.201710 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db5e569e3cd147088982678eeae71397. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:09.201817 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: This master's current role is: LEARNER
W20260928 14:48:09.270668 25870 tablet.cc:2369] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:48:09.324841 26143 tablet.cc:2369] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:09.370100 26173 raft_consensus.cc:482] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:09.370234 26173 raft_consensus.cc:504] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } }
I20260928 14:48:09.370563 26173 leader_election.cc:288] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c3b535fb9541e89be93fcb8d8155c1 (127.18.96.129:38549), 2f6fc171f9f943a7bb6100700bc6c615 (127.18.96.130:44377)
I20260928 14:48:09.374680 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.370933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:47535 (local address 127.18.96.129:38549)
0928 14:48:09.370975 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.370977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.370988 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:09.371022 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.371023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.371032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.371099 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:09.371313 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.372091 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.373752 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.374308 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.374380 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.374389 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.374562 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.374603 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:48:09.374680 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.370832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:53123 (local address 127.18.96.130:44377)
0928 14:48:09.370991 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.370994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.371003 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:09.371028 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.371030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.371039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.371099 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:09.371312 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.372135 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.373737 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.374293 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.374380 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.374389 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.374560 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.374603 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:48:09.374702 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.370747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:44377 (local address 127.18.96.131:53123)
0928 14:48:09.370926 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:09.370935 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:09.370973 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:09.371130 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:09.371130 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:09.371131 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:09.371231 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.371233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.372209 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.372209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.373563 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.373566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.374394 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.374395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.374475 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.374479 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:48:09.374538 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:48:09.374588 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":187}
I20260928 14:48:09.374715 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.370747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38549 (local address 127.18.96.131:47535)
0928 14:48:09.370819 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:09.370834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:09.370902 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:09.371126 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:09.371127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:09.371128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:09.371231 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.371233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.372205 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.372205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.373563 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:09.373566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.374394 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.374395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:09.374475 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.374479 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:48:09.374538 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:48:09.374634 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":130}
I20260928 14:48:09.375294 25958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6b45f18f4146f7807b29afce5cea0a" candidate_uuid: "c4bbac146304448287d481482d4cc6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6fc171f9f943a7bb6100700bc6c615" is_pre_election: true
I20260928 14:48:09.375270 25821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6b45f18f4146f7807b29afce5cea0a" candidate_uuid: "c4bbac146304448287d481482d4cc6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c3b535fb9541e89be93fcb8d8155c1" is_pre_election: true
I20260928 14:48:09.375448 25958 raft_consensus.cc:2454] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4bbac146304448287d481482d4cc6bb in term 0.
I20260928 14:48:09.375453 25821 raft_consensus.cc:2454] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4bbac146304448287d481482d4cc6bb in term 0.
I20260928 14:48:09.375705 26027 leader_election.cc:302] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c3b535fb9541e89be93fcb8d8155c1, c4bbac146304448287d481482d4cc6bb; no voters:
I20260928 14:48:09.375891 26173 raft_consensus.cc:2781] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:09.375980 26173 raft_consensus.cc:482] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:09.376013 26173 raft_consensus.cc:3037] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:09.376827 26173 raft_consensus.cc:504] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } }
I20260928 14:48:09.377022 26173 leader_election.cc:288] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [CANDIDATE]: Term 1 election: Requested vote from peers 02c3b535fb9541e89be93fcb8d8155c1 (127.18.96.129:38549), 2f6fc171f9f943a7bb6100700bc6c615 (127.18.96.130:44377)
I20260928 14:48:09.377177 25821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6b45f18f4146f7807b29afce5cea0a" candidate_uuid: "c4bbac146304448287d481482d4cc6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c3b535fb9541e89be93fcb8d8155c1"
I20260928 14:48:09.377269 25821 raft_consensus.cc:3037] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:09.377254 25958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6b45f18f4146f7807b29afce5cea0a" candidate_uuid: "c4bbac146304448287d481482d4cc6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6fc171f9f943a7bb6100700bc6c615"
I20260928 14:48:09.377348 25958 raft_consensus.cc:3037] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:09.378014 25821 raft_consensus.cc:2454] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4bbac146304448287d481482d4cc6bb in term 1.
I20260928 14:48:09.378234 26027 leader_election.cc:302] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c3b535fb9541e89be93fcb8d8155c1, c4bbac146304448287d481482d4cc6bb; no voters:
I20260928 14:48:09.378245 25958 raft_consensus.cc:2454] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4bbac146304448287d481482d4cc6bb in term 1.
I20260928 14:48:09.378345 26173 raft_consensus.cc:2781] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:09.378506 26173 raft_consensus.cc:686] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 1 LEADER]: Becoming Leader. State: Replica: c4bbac146304448287d481482d4cc6bb, State: Running, Role: LEADER
I20260928 14:48:09.378610 26173 consensus_queue.cc:237] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } }
I20260928 14:48:09.379534 24905 catalog_manager.cc:5495] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb reported cstate change: term changed from 0 to 1, leader changed from <none> to c4bbac146304448287d481482d4cc6bb (127.18.96.131). New cstate: current_term: 1 leader_uuid: "c4bbac146304448287d481482d4cc6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } health_report { overall_health: UNKNOWN } } }
I20260928 14:48:09.380208 26180 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: NON_VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: true } } }
I20260928 14:48:09.380312 26180 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:09.381024 26180 raft_consensus.cc:1050] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: attempting to promote NON_VOTER a2445c83ebe6491aacbb425ac17eefe6 to VOTER
I20260928 14:48:09.381163 26180 consensus_queue.cc:237] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } }
I20260928 14:48:09.381614 25698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 LEARNER]: Refusing update from remote peer db5e569e3cd147088982678eeae71397: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 14:48:09.381803 26180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 14:48:09.383528 25511 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Refusing update from remote peer db5e569e3cd147088982678eeae71397: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 14:48:09.383801 25660 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 14:48:09.384106 25463 raft_consensus.cc:2932] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a2445c83ebe6491aacbb425ac17eefe6 (127.18.96.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:09.384315 25698 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a2445c83ebe6491aacbb425ac17eefe6 (127.18.96.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:09.384541 25511 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a2445c83ebe6491aacbb425ac17eefe6 (127.18.96.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:09.384986 25463 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:09.385054 25463 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:09.385159 25463 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:09.385169 25660 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:09.385224 25463 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:09.385243 25660 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:09.385447 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:09.385547 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:09.385824 26182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db5e569e3cd147088982678eeae71397" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:09.385972 26182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:09.395362 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.392662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.18.96.131:40059)
0928 14:48:09.392730 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.392733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.392742 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:09.392782 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.392783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.392805 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.392894 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:09.393063 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.393600 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.394470 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.394715 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.394807 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.394995 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:09.395122 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:09.395227 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.395247 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.395270 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:48:09.397423 25821 raft_consensus.cc:1261] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [term 1 FOLLOWER]: Refusing update from remote peer c4bbac146304448287d481482d4cc6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:09.397440 25958 raft_consensus.cc:1261] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [term 1 FOLLOWER]: Refusing update from remote peer c4bbac146304448287d481482d4cc6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:09.397832 26178 consensus_queue.cc:1035] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:09.398010 26173 consensus_queue.cc:1035] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:09.401300 26185 mvcc.cc:205] Tried to move back new op lower bound from 7334325818968915968 to 7334325818895368192. Current Snapshot: MvccSnapshot[applied={T|T < 7334325818968915968 or (T in {7334325818968915968})}]
I20260928 14:48:09.401860 26186 mvcc.cc:205] Tried to move back new op lower bound from 7334325818968915968 to 7334325818895368192. Current Snapshot: MvccSnapshot[applied={T|T < 7334325818968915968}]
I20260928 14:48:09.405061 24905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:09.405164 24905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:09.406525 24905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db5e569e3cd147088982678eeae71397: Sending DeleteTablet for 3 replicas of tablet 0c6b45f18f4146f7807b29afce5cea0a
I20260928 14:48:09.406853 25801 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6b45f18f4146f7807b29afce5cea0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:09 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:56234
I20260928 14:48:09.406862 25938 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6b45f18f4146f7807b29afce5cea0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:09 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:35282
I20260928 14:48:09.407092 26194 tablet_replica.cc:330] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: stopping tablet replica
I20260928 14:48:09.407083 26193 tablet_replica.cc:330] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: stopping tablet replica
I20260928 14:48:09.407188 26194 raft_consensus.cc:2229] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:09.407199 26193 raft_consensus.cc:2229] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:09.407290 26194 raft_consensus.cc:2258] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:09.407308 26193 raft_consensus.cc:2258] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:09.407433 26074 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6b45f18f4146f7807b29afce5cea0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:09 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:39070
I20260928 14:48:09.407830 26193 ts_tablet_manager.cc:1905] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:09.408354 26194 ts_tablet_manager.cc:1905] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:09.408974 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 24873
I20260928 14:48:09.410295 26193 ts_tablet_manager.cc:1918] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:09.410368 26193 log.cc:1196] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/0c6b45f18f4146f7807b29afce5cea0a
I20260928 14:48:09.410633 26193 ts_tablet_manager.cc:1939] T 0c6b45f18f4146f7807b29afce5cea0a P 02c3b535fb9541e89be93fcb8d8155c1: Deleting consensus metadata
W20260928 14:48:09.417102 25757 connection.cc:535] server connection from 127.0.0.1:56234 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:48:09.417554 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.408375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.18.96.188:37387)
0928 14:48:09.408418 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.408420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.408975 (+ 555us) server_negotiation.cc:407] Connection header received
0928 14:48:09.409071 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.409072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.409082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.409109 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:09.410367 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.411132 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.412237 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.412581 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.412653 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.414335 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:09.414336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:09.414337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:09.414344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:09.416215 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.416619 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.416621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.416621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.416673 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.416873 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.416874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.416874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.417035 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:09.417112 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.417337 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.417424 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:48:09.418259 26194 ts_tablet_manager.cc:1918] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:09.418351 26194 log.cc:1196] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/0c6b45f18f4146f7807b29afce5cea0a
I20260928 14:48:09.418696 26194 ts_tablet_manager.cc:1939] T 0c6b45f18f4146f7807b29afce5cea0a P 2f6fc171f9f943a7bb6100700bc6c615: Deleting consensus metadata
I20260928 14:48:09.419073 26195 tablet_replica.cc:330] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: stopping tablet replica
W20260928 14:48:09.419077 25893 connection.cc:441] server connection from 127.0.0.1:35282 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35282 (request call id 1) could send its response
I20260928 14:48:09.419255 26195 raft_consensus.cc:2229] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:48:09.419446 26195 raft_consensus.cc:2258] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:09.420049 26195 ts_tablet_manager.cc:1905] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:09.421486 26195 ts_tablet_manager.cc:1918] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:09.421559 26195 log.cc:1196] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/0c6b45f18f4146f7807b29afce5cea0a
I20260928 14:48:09.421818 26195 ts_tablet_manager.cc:1939] T 0c6b45f18f4146f7807b29afce5cea0a P c4bbac146304448287d481482d4cc6bb: Deleting consensus metadata
W20260928 14:48:09.422331 26029 connection.cc:441] server connection from 127.0.0.1:39070 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39070 (request call id 1) could send its response
I20260928 14:48:09.427641 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.419566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.18.96.188:37387)
0928 14:48:09.419602 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.419604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.419949 (+ 345us) server_negotiation.cc:407] Connection header received
0928 14:48:09.420048 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.420049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.420058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.420087 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:09.420985 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.421508 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.422819 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.423029 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.423108 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.424659 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:09.424660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:09.424661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:09.424667 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:09.426173 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.426566 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.426567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.426567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.426634 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.426872 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.426872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.426873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.426997 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:09.427095 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.427461 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.427539 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:48:09.427650 26153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:09.419362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.18.96.189:42805)
0928 14:48:09.419418 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:09.419420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:09.419525 (+ 105us) server_negotiation.cc:407] Connection header received
0928 14:48:09.419630 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:09.419631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:09.419647 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:09.419682 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:09.420835 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.421564 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.422694 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:09.423016 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:09.423084 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:09.424691 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:09.424692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:09.424692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:09.424699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:09.426438 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.426817 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.426818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.426819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.426885 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:09.427075 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:09.427076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:09.427077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:09.427189 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:09.427286 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:09.427435 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 14:48:09.427531 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:48:10.402107 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.401818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.129:42775)
0928 14:48:10.401969 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.402053 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
W20260928 14:48:10.402416 25868 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:39113 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
I20260928 14:48:10.402832 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.402534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.131:49649)
0928 14:48:10.402729 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.402791 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":61,"threads_started":1}
I20260928 14:48:10.402832 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.402534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.130:49979)
0928 14:48:10.402724 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.402791 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":53,"threads_started":1}
W20260928 14:48:10.403002 26005 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:39113 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
W20260928 14:48:10.403093 26141 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:39113 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
I20260928 14:48:10.907384 26207 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db5e569e3cd147088982678eeae71397)
I20260928 14:48:10.907616 26207 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } }
I20260928 14:48:10.907764 26208 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db5e569e3cd147088982678eeae71397)
I20260928 14:48:10.907931 26208 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } }
I20260928 14:48:10.908094 26207 leader_election.cc:288] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db5e569e3cd147088982678eeae71397 (127.18.96.190:39113), a2445c83ebe6491aacbb425ac17eefe6 (127.18.96.188:37387)
I20260928 14:48:10.908190 26208 leader_election.cc:288] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db5e569e3cd147088982678eeae71397 (127.18.96.190:39113), ec8f238e20044baf9d7ce74d06e053a2 (127.18.96.189:42805)
I20260928 14:48:10.908454 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.908222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42858)
0928 14:48:10.908343 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.908405 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260928 14:48:10.908619 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.908296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42860)
0928 14:48:10.908514 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.908575 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":46,"threads_started":1}
W20260928 14:48:10.908672 25486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
W20260928 14:48:10.908855 25672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
I20260928 14:48:10.909127 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.909030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42870)
0928 14:48:10.909067 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.909093 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 14:48:10.909258 25486 leader_election.cc:334] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db5e569e3cd147088982678eeae71397 (127.18.96.190:39113): Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
I20260928 14:48:10.909268 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.909177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42884)
0928 14:48:10.909218 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.909241 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 14:48:10.909487 25672 leader_election.cc:334] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db5e569e3cd147088982678eeae71397 (127.18.96.190:39113): Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
I20260928 14:48:10.911466 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.908551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58178 (local address 127.18.96.188:37387)
0928 14:48:10.908619 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:10.908620 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:10.908628 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:48:10.908657 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:10.908658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:10.908669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:10.908904 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:10.908940 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.909758 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.910731 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.911110 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.911196 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.911208 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:10.911351 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 14:48:10.911393 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"thread_start_us":39,"threads_started":1}
I20260928 14:48:10.911459 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.908290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:37387 (local address 127.0.0.1:58178)
0928 14:48:10.908445 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.908457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:10.908516 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:10.908766 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:10.908768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:10.908770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:10.908842 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.908844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.909845 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.909846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.910624 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.910626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.911195 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.911196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.911262 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.911265 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:10.911343 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:48:10.911393 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":60,"threads_started":1}
I20260928 14:48:10.911705 25698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec8f238e20044baf9d7ce74d06e053a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a2445c83ebe6491aacbb425ac17eefe6" is_pre_election: true
I20260928 14:48:10.911805 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.908650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.18.96.189:42805)
0928 14:48:10.908813 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:10.908815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:10.908824 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:10.909153 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:10.909155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:10.909175 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:10.909230 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:10.909443 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.909973 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.910931 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.911296 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.911406 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.911417 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:10.911680 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 14:48:10.911726 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":150,"thread_start_us":37,"threads_started":1}
I20260928 14:48:10.911813 25698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8f238e20044baf9d7ce74d06e053a2 in term 1.
I20260928 14:48:10.912014 25489 leader_election.cc:302] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2445c83ebe6491aacbb425ac17eefe6, ec8f238e20044baf9d7ce74d06e053a2; no voters: db5e569e3cd147088982678eeae71397
I20260928 14:48:10.912072 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.908473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42805 (local address 127.0.0.1:43530)
0928 14:48:10.908729 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.908740 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:10.909061 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:10.909301 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:10.909302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:10.909304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:10.909357 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.909359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.910056 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.910057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.910815 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.910818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.911417 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.911419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.911484 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.911487 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:10.911941 (+ 454us) client_negotiation.cc:240] Negotiation successful
0928 14:48:10.911991 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":51,"threads_started":1}
I20260928 14:48:10.912140 26207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:48:10.912191 26207 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader db5e569e3cd147088982678eeae71397)
I20260928 14:48:10.912210 26207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:48:10.912253 25511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2445c83ebe6491aacbb425ac17eefe6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec8f238e20044baf9d7ce74d06e053a2" is_pre_election: true
I20260928 14:48:10.913187 26207 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } }
I20260928 14:48:10.913318 25511 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2445c83ebe6491aacbb425ac17eefe6 in current term 2: Already voted for candidate ec8f238e20044baf9d7ce74d06e053a2 in this term.
I20260928 14:48:10.913332 26207 leader_election.cc:288] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [CANDIDATE]: Term 2 election: Requested vote from peers db5e569e3cd147088982678eeae71397 (127.18.96.190:39113), a2445c83ebe6491aacbb425ac17eefe6 (127.18.96.188:37387)
I20260928 14:48:10.913515 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.913394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42904)
0928 14:48:10.913469 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.913493 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:48:10.913479 25698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec8f238e20044baf9d7ce74d06e053a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a2445c83ebe6491aacbb425ac17eefe6"
I20260928 14:48:10.913553 25698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:48:10.913864 25674 leader_election.cc:302] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2445c83ebe6491aacbb425ac17eefe6; no voters: db5e569e3cd147088982678eeae71397, ec8f238e20044baf9d7ce74d06e053a2
I20260928 14:48:10.914395 25698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8f238e20044baf9d7ce74d06e053a2 in term 2.
I20260928 14:48:10.914574 25489 leader_election.cc:302] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2445c83ebe6491aacbb425ac17eefe6, ec8f238e20044baf9d7ce74d06e053a2; no voters:
I20260928 14:48:10.914606 26208 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:48:10.914650 26207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:48:10.914719 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.914620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42920)
0928 14:48:10.914670 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.914693 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 14:48:10.914793 26207 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [term 2 LEADER]: Becoming Leader. State: Replica: ec8f238e20044baf9d7ce74d06e053a2, State: Running, Role: LEADER
I20260928 14:48:10.914866 26207 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } }
I20260928 14:48:10.915220 26217 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec8f238e20044baf9d7ce74d06e053a2. Latest consensus state: current_term: 2 leader_uuid: "ec8f238e20044baf9d7ce74d06e053a2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:10.915323 26217 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: This master's current role is: LEADER
W20260928 14:48:10.915530 25486 leader_election.cc:334] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db5e569e3cd147088982678eeae71397 (127.18.96.190:39113): Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
I20260928 14:48:10.915530 26219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:10.916072 26219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:10.916132 26219 catalog_manager.cc:1259] Loaded cluster ID: 0c27a911cfc5475a9bc64e29ac98a8b4
I20260928 14:48:10.916172 26219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:10.916340 26219 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:10.916399 26219 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Loaded TSK: 0
I20260928 14:48:10.916559 26219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:48:10.972985 25501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:10.974694 25501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:10.975800 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.975646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42928)
0928 14:48:10.975728 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.975767 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 14:48:10.975805 25698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [term 2 FOLLOWER]: Refusing update from remote peer ec8f238e20044baf9d7ce74d06e053a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 14:48:10.976236 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.976109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.0.0.1:42940)
0928 14:48:10.976139 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.976193 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 14:48:10.976181 26217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 14:48:10.976323 25486 consensus_peers.cc:487] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 -> Peer db5e569e3cd147088982678eeae71397 (127.18.96.190:39113): Couldn't send request to peer db5e569e3cd147088982678eeae71397. Status: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:48:10.976797 26208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec8f238e20044baf9d7ce74d06e053a2. Latest consensus state: current_term: 2 leader_uuid: "ec8f238e20044baf9d7ce74d06e053a2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:10.976902 26208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:10.977321 26220 mvcc.cc:205] Tried to move back new op lower bound from 7334325825434804224 to 7334325825187889152. Current Snapshot: MvccSnapshot[applied={T|T < 7334325825434804224}]
I20260928 14:48:10.977450 26217 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ec8f238e20044baf9d7ce74d06e053a2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:10.977540 26217 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:10.977620 26221 mvcc.cc:205] Tried to move back new op lower bound from 7334325825434804224 to 7334325825187889152. Current Snapshot: MvccSnapshot[applied={T|T < 7334325825434804224}]
I20260928 14:48:10.977705 26208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ec8f238e20044baf9d7ce74d06e053a2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "db5e569e3cd147088982678eeae71397" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 39113 } } peers { permanent_uuid: "ec8f238e20044baf9d7ce74d06e053a2" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42805 } attrs { promote: false } } peers { permanent_uuid: "a2445c83ebe6491aacbb425ac17eefe6" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 37387 } attrs { promote: false } } }
I20260928 14:48:10.977800 26208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2445c83ebe6491aacbb425ac17eefe6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:10.982805 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.979561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:44377 (local address 127.0.0.1:35284)
0928 14:48:10.980060 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.980069 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:10.980120 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:10.980305 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:10.980305 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:10.980308 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:10.980367 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.980369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.981001 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.981002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.981753 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.981755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.982386 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.982387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.982637 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.982642 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:48:10.982693 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:48:10.982736 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":354,"thread_start_us":87,"threads_started":1}
I20260928 14:48:10.983048 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.979647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.18.96.130:44377)
0928 14:48:10.979941 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:10.979952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:10.980113 (+ 161us) server_negotiation.cc:407] Connection header received
0928 14:48:10.980185 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:10.980186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:10.980197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:10.980253 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:10.980434 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.980915 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.981857 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.982286 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.982381 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.982394 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:10.982923 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 14:48:10.982981 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":81,"threads_started":1}
I20260928 14:48:10.983105 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.979620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38549 (local address 127.0.0.1:56246)
0928 14:48:10.979766 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.979779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:10.979830 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:10.980012 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:10.980013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:10.980016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:10.980071 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.980072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.980956 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.980957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.982134 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.982136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.982883 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.982884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.982943 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.982946 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:10.982986 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 14:48:10.983038 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 14:48:10.983254 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.979751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.18.96.129:38549)
0928 14:48:10.979860 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:10.979893 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:10.979904 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:10.979932 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:10.979933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:10.979946 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:10.980012 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:10.980123 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.980865 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.982234 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.982802 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.982869 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.982972 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:10.983000 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 14:48:10.983040 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260928 14:48:10.983351 25938 tablet_service.cc:1467] Processing CreateTablet for tablet f0bfbf9374bb444a81d9de737ebb594e (DEFAULT_TABLE table=test-table [id=7ede4df5f8134a53b970be9513e43cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:10.983397 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.979555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:40059 (local address 127.0.0.1:39086)
0928 14:48:10.979589 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.979600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:10.979639 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:10.980355 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:10.980355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:10.980357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:10.980391 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.980392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.981271 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.981271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.982312 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:10.982313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.983166 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.983167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:10.983238 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.983243 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:48:10.983272 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 14:48:10.983326 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 14:48:10.983487 25938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0bfbf9374bb444a81d9de737ebb594e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:10.983700 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.979703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.18.96.131:40059)
0928 14:48:10.980221 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:10.980223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:10.980231 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:48:10.980257 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:10.980258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:10.980268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:10.980332 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:10.980480 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.981187 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.982468 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:10.983068 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:10.983571 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:10.983586 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:10.983607 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 14:48:10.983649 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":59,"threads_started":1}
I20260928 14:48:10.983963 26074 tablet_service.cc:1467] Processing CreateTablet for tablet f0bfbf9374bb444a81d9de737ebb594e (DEFAULT_TABLE table=test-table [id=7ede4df5f8134a53b970be9513e43cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:10.984057 25801 tablet_service.cc:1467] Processing CreateTablet for tablet f0bfbf9374bb444a81d9de737ebb594e (DEFAULT_TABLE table=test-table [id=7ede4df5f8134a53b970be9513e43cdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:10.984095 26074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0bfbf9374bb444a81d9de737ebb594e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:10.984211 25801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0bfbf9374bb444a81d9de737ebb594e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:10.985037 26229 tablet_bootstrap.cc:492] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: Bootstrap starting.
I20260928 14:48:10.985492 26229 tablet_bootstrap.cc:654] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:10.985633 26230 tablet_bootstrap.cc:492] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: Bootstrap starting.
I20260928 14:48:10.985762 26231 tablet_bootstrap.cc:492] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1: Bootstrap starting.
I20260928 14:48:10.986009 26230 tablet_bootstrap.cc:654] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:10.986035 26231 tablet_bootstrap.cc:654] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:10.986291 26229 tablet_bootstrap.cc:492] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: No bootstrap required, opened a new log
I20260928 14:48:10.986344 26229 ts_tablet_manager.cc:1397] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:10.986505 26229 raft_consensus.cc:348] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } }
I20260928 14:48:10.986594 26229 raft_consensus.cc:374] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:10.986621 26229 raft_consensus.cc:729] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6fc171f9f943a7bb6100700bc6c615, State: Initialized, Role: FOLLOWER
I20260928 14:48:10.986711 26231 tablet_bootstrap.cc:492] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1: No bootstrap required, opened a new log
I20260928 14:48:10.986697 26229 consensus_queue.cc:260] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } }
I20260928 14:48:10.986759 26231 ts_tablet_manager.cc:1397] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:10.986871 26229 ts_tablet_manager.cc:1428] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:10.986927 26231 raft_consensus.cc:348] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } }
I20260928 14:48:10.987000 26231 raft_consensus.cc:374] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:10.987059 26231 raft_consensus.cc:729] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c3b535fb9541e89be93fcb8d8155c1, State: Initialized, Role: FOLLOWER
I20260928 14:48:10.987129 26231 consensus_queue.cc:260] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } }
I20260928 14:48:10.987293 26231 ts_tablet_manager.cc:1428] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:10.987711 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.987502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.129:37559)
0928 14:48:10.987630 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.987668 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":89,"threads_started":1}
I20260928 14:48:10.988047 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.987965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.129:55937)
0928 14:48:10.988001 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.988025 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 14:48:10.988184 26006 heartbeater.cc:503] Master 127.18.96.189:42805 was elected leader, sending a full tablet report...
I20260928 14:48:10.988334 25869 heartbeater.cc:503] Master 127.18.96.189:42805 was elected leader, sending a full tablet report...
I20260928 14:48:10.988674 26230 tablet_bootstrap.cc:492] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: No bootstrap required, opened a new log
I20260928 14:48:10.988731 26230 ts_tablet_manager.cc:1397] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:48:10.988847 26230 raft_consensus.cc:348] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } }
I20260928 14:48:10.988919 26230 raft_consensus.cc:374] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:10.988933 26230 raft_consensus.cc:729] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4bbac146304448287d481482d4cc6bb, State: Initialized, Role: FOLLOWER
I20260928 14:48:10.988966 26230 consensus_queue.cc:260] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } }
I20260928 14:48:10.989099 26230 ts_tablet_manager.cc:1428] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:10.989403 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.987674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.130:55743)
0928 14:48:10.989317 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.989362 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1617,"thread_start_us":218,"threads_started":1}
I20260928 14:48:10.989552 26142 heartbeater.cc:503] Master 127.18.96.189:42805 was elected leader, sending a full tablet report...
I20260928 14:48:10.989754 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.989678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.130:58039)
0928 14:48:10.989709 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.989733 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 14:48:10.989972 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.989295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.131:51561)
0928 14:48:10.989903 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.989942 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":86,"threads_started":1}
I20260928 14:48:10.990342 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:10.990222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.131:44985)
0928 14:48:10.990291 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:10.990321 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 14:48:11.062338 26236 raft_consensus.cc:482] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:11.062527 26236 raft_consensus.cc:504] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } }
I20260928 14:48:11.062990 26236 leader_election.cc:288] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f6fc171f9f943a7bb6100700bc6c615 (127.18.96.130:44377), c4bbac146304448287d481482d4cc6bb (127.18.96.131:40059)
I20260928 14:48:11.066465 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.063284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:43987 (local address 127.18.96.131:40059)
0928 14:48:11.063347 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:11.063349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:11.063451 (+ 102us) server_negotiation.cc:407] Connection header received
0928 14:48:11.063536 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:11.063537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:11.063547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:11.063607 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:11.063793 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.064278 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.065709 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.066117 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.066171 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:11.066179 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:11.066346 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:48:11.066391 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:48:11.066629 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.063166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:44377 (local address 127.18.96.129:49459)
0928 14:48:11.063224 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.063236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:11.063312 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:11.063543 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:11.063544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:11.063547 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:11.063636 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:11.063638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.064355 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.064356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:11.065680 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:11.065683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.066378 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.066379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:11.066444 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:11.066452 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:48:11.066512 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 14:48:11.066556 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 14:48:11.066658 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.063227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:49459 (local address 127.18.96.130:44377)
0928 14:48:11.063273 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:11.063275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:11.063286 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:11.063452 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:11.063453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:11.063465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:11.063523 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:11.063717 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.064269 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.065758 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.066277 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.066340 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:11.066349 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:11.066537 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 14:48:11.066598 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:48:11.066471 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.063214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:40059 (local address 127.18.96.129:43987)
0928 14:48:11.063378 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.063391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:11.063459 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:11.063633 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:11.063634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:11.063635 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:11.063713 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:11.063715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.064384 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.064385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:11.065613 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:11.065616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.066187 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.066188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:11.066261 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:11.066265 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:48:11.066330 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:48:11.066375 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260928 14:48:11.066855 25958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bfbf9374bb444a81d9de737ebb594e" candidate_uuid: "02c3b535fb9541e89be93fcb8d8155c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6fc171f9f943a7bb6100700bc6c615" is_pre_election: true
I20260928 14:48:11.066978 25958 raft_consensus.cc:2454] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c3b535fb9541e89be93fcb8d8155c1 in term 0.
I20260928 14:48:11.067124 26094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bfbf9374bb444a81d9de737ebb594e" candidate_uuid: "02c3b535fb9541e89be93fcb8d8155c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4bbac146304448287d481482d4cc6bb" is_pre_election: true
I20260928 14:48:11.067176 25756 leader_election.cc:302] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c3b535fb9541e89be93fcb8d8155c1, 2f6fc171f9f943a7bb6100700bc6c615; no voters:
I20260928 14:48:11.067281 26094 raft_consensus.cc:2454] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c3b535fb9541e89be93fcb8d8155c1 in term 0.
I20260928 14:48:11.067412 26236 raft_consensus.cc:2781] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:11.067477 26236 raft_consensus.cc:482] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:11.067502 26236 raft_consensus.cc:3037] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:11.068317 26236 raft_consensus.cc:504] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } }
I20260928 14:48:11.068475 26236 leader_election.cc:288] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [CANDIDATE]: Term 1 election: Requested vote from peers 2f6fc171f9f943a7bb6100700bc6c615 (127.18.96.130:44377), c4bbac146304448287d481482d4cc6bb (127.18.96.131:40059)
I20260928 14:48:11.068662 26094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bfbf9374bb444a81d9de737ebb594e" candidate_uuid: "02c3b535fb9541e89be93fcb8d8155c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4bbac146304448287d481482d4cc6bb"
I20260928 14:48:11.068662 25958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bfbf9374bb444a81d9de737ebb594e" candidate_uuid: "02c3b535fb9541e89be93fcb8d8155c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6fc171f9f943a7bb6100700bc6c615"
I20260928 14:48:11.068737 25958 raft_consensus.cc:3037] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:11.068737 26094 raft_consensus.cc:3037] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:11.069486 25958 raft_consensus.cc:2454] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c3b535fb9541e89be93fcb8d8155c1 in term 1.
I20260928 14:48:11.069551 26094 raft_consensus.cc:2454] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c3b535fb9541e89be93fcb8d8155c1 in term 1.
I20260928 14:48:11.069666 25756 leader_election.cc:302] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c3b535fb9541e89be93fcb8d8155c1, 2f6fc171f9f943a7bb6100700bc6c615; no voters:
I20260928 14:48:11.069756 26236 raft_consensus.cc:2781] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:11.069922 26236 raft_consensus.cc:686] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [term 1 LEADER]: Becoming Leader. State: Replica: 02c3b535fb9541e89be93fcb8d8155c1, State: Running, Role: LEADER
I20260928 14:48:11.070091 26236 consensus_queue.cc:237] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } }
I20260928 14:48:11.070282 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.070187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.129:55971)
0928 14:48:11.070222 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.070252 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 14:48:11.070922 25501 catalog_manager.cc:5495] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c3b535fb9541e89be93fcb8d8155c1 (127.18.96.129). New cstate: current_term: 1 leader_uuid: "02c3b535fb9541e89be93fcb8d8155c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02c3b535fb9541e89be93fcb8d8155c1" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38549 } health_report { overall_health: HEALTHY } } }
I20260928 14:48:11.077157 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.074619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.18.96.129:38549)
0928 14:48:11.074915 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:11.074917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:11.074924 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:48:11.074952 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:11.074953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:11.074968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:11.075032 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:11.075176 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.075638 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.076444 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:11.076648 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:11.076702 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:11.076849 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:11.076952 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:11.077007 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:11.077048 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 14:48:11.077066 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260928 14:48:11.078228 25958 raft_consensus.cc:1261] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [term 1 FOLLOWER]: Refusing update from remote peer 02c3b535fb9541e89be93fcb8d8155c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:11.078478 26094 raft_consensus.cc:1261] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [term 1 FOLLOWER]: Refusing update from remote peer 02c3b535fb9541e89be93fcb8d8155c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:11.078482 26242 consensus_queue.cc:1035] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6fc171f9f943a7bb6100700bc6c615" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:11.078713 26242 consensus_queue.cc:1035] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4bbac146304448287d481482d4cc6bb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:11.079164 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.079046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.130:57405)
0928 14:48:11.079092 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.079132 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 14:48:11.079542 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.079441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.131:50647)
0928 14:48:11.079474 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.079509 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 14:48:11.080772 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.080620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.129:43777)
0928 14:48:11.080709 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.080744 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:48:11.081004 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.080762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.131:40863)
0928 14:48:11.080943 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.080972 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260928 14:48:11.081538 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.081423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.130:36737)
0928 14:48:11.081486 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.081510 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:48:11.082022 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.081897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:39113 (local address 127.18.96.129:34347)
0928 14:48:11.081958 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.081992 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 14:48:11.083249 26248 mvcc.cc:205] Tried to move back new op lower bound from 7334325825854238720 to 7334325825823629312. Current Snapshot: MvccSnapshot[applied={T|T < 7334325825854238720 or (T in {7334325825854238720})}]
I20260928 14:48:11.083314 25502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:11.083412 25502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:11.084794 25502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec8f238e20044baf9d7ce74d06e053a2: Sending DeleteTablet for 3 replicas of tablet f0bfbf9374bb444a81d9de737ebb594e
I20260928 14:48:11.085222 25801 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bfbf9374bb444a81d9de737ebb594e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:11 UTC) from {username='oryx', principal='oryx/127.18.96.189@KRBTEST.COM'} at 127.0.0.1:56246
I20260928 14:48:11.085301 25938 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bfbf9374bb444a81d9de737ebb594e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:11 UTC) from {username='oryx', principal='oryx/127.18.96.189@KRBTEST.COM'} at 127.0.0.1:35284
I20260928 14:48:11.085440 26074 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bfbf9374bb444a81d9de737ebb594e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:11 UTC) from {username='oryx', principal='oryx/127.18.96.189@KRBTEST.COM'} at 127.0.0.1:39086
I20260928 14:48:11.085636 26257 tablet_replica.cc:330] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: stopping tablet replica
I20260928 14:48:11.085784 26257 raft_consensus.cc:2229] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:11.085880 26257 raft_consensus.cc:2258] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:11.085922 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 25649
I20260928 14:48:11.085850 26255 tablet_replica.cc:330] T f0bfbf9374bb444a81d9de737ebb594e P 02c3b535fb9541e89be93fcb8d8155c1: stopping tablet replica
I20260928 14:48:11.086944 26256 tablet_replica.cc:330] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: stopping tablet replica
I20260928 14:48:11.086999 26256 raft_consensus.cc:2229] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:11.087042 26256 raft_consensus.cc:2258] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:11.087453 26256 ts_tablet_manager.cc:1905] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:11.088099 26257 ts_tablet_manager.cc:1905] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:11.089521 26256 ts_tablet_manager.cc:1918] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:11.089586 26256 log.cc:1196] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/f0bfbf9374bb444a81d9de737ebb594e
I20260928 14:48:11.089612 26257 ts_tablet_manager.cc:1918] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:11.089673 26257 log.cc:1196] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/f0bfbf9374bb444a81d9de737ebb594e
I20260928 14:48:11.089838 26256 ts_tablet_manager.cc:1939] T f0bfbf9374bb444a81d9de737ebb594e P 2f6fc171f9f943a7bb6100700bc6c615: Deleting consensus metadata
I20260928 14:48:11.089950 26257 ts_tablet_manager.cc:1939] T f0bfbf9374bb444a81d9de737ebb594e P c4bbac146304448287d481482d4cc6bb: Deleting consensus metadata
I20260928 14:48:11.090279 25489 catalog_manager.cc:4841] TS c4bbac146304448287d481482d4cc6bb (127.18.96.131:40059): tablet f0bfbf9374bb444a81d9de737ebb594e (table test-table [id=7ede4df5f8134a53b970be9513e43cdf]) successfully deleted
I20260928 14:48:11.090404 25488 catalog_manager.cc:4841] TS 2f6fc171f9f943a7bb6100700bc6c615 (127.18.96.130:44377): tablet f0bfbf9374bb444a81d9de737ebb594e (table test-table [id=7ede4df5f8134a53b970be9513e43cdf]) successfully deleted
I20260928 14:48:11.092218 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:11.092076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38549 (local address 127.0.0.1:56276)
0928 14:48:11.092122 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:11.092174 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:38549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 14:48:11.092422 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 25875
W20260928 14:48:11.092410 25486 catalog_manager.cc:4584] TS 02c3b535fb9541e89be93fcb8d8155c1 (127.18.96.129:38549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0bfbf9374bb444a81d9de737ebb594e: Network error: Client connection negotiation failed: client connection to 127.18.96.129:38549: connect: Connection refused (error 111)
I20260928 14:48:11.097366 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 26011
I20260928 14:48:11.103113 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 25369
I20260928 14:48:11.108173 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 25556
2026-09-28T14:48:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4908 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:11 dist-test-slave-j968 krb5kdc[26260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:11 dist-test-slave-j968 krb5kdc[26260](info): set up 2 sockets
Sep 28 14:48:11 dist-test-slave-j968 krb5kdc[26260](info): commencing operation
krb5kdc: starting...
W20260928 14:48:12.177477 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:48:12 dist-test-slave-j968 krb5kdc[26260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.190.keytab.
Entry for principal oryx/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.190.keytab.
I20260928 14:48:12.307725 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:42625
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.190.keytab
--principal=oryx/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:12.396898 26273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:12.397125 26273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:12.397148 26273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:12.398383 26273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:12.398427 26273 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:48:12.398442 26273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:12.398454 26273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:12.398511 26273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:12.398530 26273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:12.400182 26273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:42625
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.190.keytab
--principal=oryx/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:42625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:12.400481 26273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:12.400731 26273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:12.406070 26279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:12.406116 26278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:12.406347 26273 server_base.cc:1034] running on GCE node
W20260928 14:48:12.406222 26281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:12.406767 26273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:12.407106 26273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:12.408278 26273 hybrid_clock.cc:648] HybridClock initialized: now 1790606892408269 us; error 35 us; skew 500 ppm
Sep 28 14:48:12 dist-test-slave-j968 krb5kdc[26260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:12.416977 26273 init.cc:377] Logged in from keytab as oryx/127.18.96.190@KRBTEST.COM (short username oryx)
I20260928 14:48:12.417520 26273 webserver.cc:460] Webserver started at http://127.18.96.190:44895/ using document root <none> and password file <none>
I20260928 14:48:12.417762 26273 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:12.417832 26273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:12.417997 26273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:12.419132 26273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "bb0c7cd12e194907976a69ecedfe9de7"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "801ae1068b6ab5579d8ab360f22f51d4"
server_key_iv: "90231533e3be82f2aec41621948212f4"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.419513 26273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "bb0c7cd12e194907976a69ecedfe9de7"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "801ae1068b6ab5579d8ab360f22f51d4"
server_key_iv: "90231533e3be82f2aec41621948212f4"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.421072 26273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:12.421979 26288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:12.422366 26273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:12.422520 26273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "bb0c7cd12e194907976a69ecedfe9de7"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "801ae1068b6ab5579d8ab360f22f51d4"
server_key_iv: "90231533e3be82f2aec41621948212f4"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.422613 26273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:12.437328 26273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:12.440093 26273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:12.440290 26273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:12.445713 26273 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:42625
I20260928 14:48:12.445765 26350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:42625 every 8 connection(s)
I20260928 14:48:12.446182 26273 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:12.447153 26351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:12.449546 26351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7: Bootstrap starting.
I20260928 14:48:12.450189 26351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:12.450503 26351 log.cc:824] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:12.451254 26351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7: No bootstrap required, opened a new log
I20260928 14:48:12.453356 26351 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0c7cd12e194907976a69ecedfe9de7" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42625 } }
I20260928 14:48:12.453526 26351 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:12.453574 26351 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0c7cd12e194907976a69ecedfe9de7, State: Initialized, Role: FOLLOWER
I20260928 14:48:12.453680 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 26273
I20260928 14:48:12.453783 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:12.453764 26351 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0c7cd12e194907976a69ecedfe9de7" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42625 } }
I20260928 14:48:12.453855 26351 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:12.453891 26351 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:12.453996 18818 external_mini_cluster.cc:1428] Setting key aa30cb2ca1409f7db7a0994ad8057bfe
I20260928 14:48:12.454037 26351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:12.454842 26351 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0c7cd12e194907976a69ecedfe9de7" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42625 } }
I20260928 14:48:12.454952 26351 leader_election.cc:302] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0c7cd12e194907976a69ecedfe9de7; no voters:
I20260928 14:48:12.455108 26351 leader_election.cc:288] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:12.455157 26356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:12.455343 26356 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [term 1 LEADER]: Becoming Leader. State: Replica: bb0c7cd12e194907976a69ecedfe9de7, State: Running, Role: LEADER
I20260928 14:48:12.455466 26351 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:12.455520 26356 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0c7cd12e194907976a69ecedfe9de7" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42625 } }
I20260928 14:48:12.455888 26356 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0c7cd12e194907976a69ecedfe9de7. Latest consensus state: current_term: 1 leader_uuid: "bb0c7cd12e194907976a69ecedfe9de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0c7cd12e194907976a69ecedfe9de7" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42625 } } }
I20260928 14:48:12.455988 26356 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:12.455896 26357 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb0c7cd12e194907976a69ecedfe9de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0c7cd12e194907976a69ecedfe9de7" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42625 } } }
I20260928 14:48:12.456359 26357 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:12.456488 26362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:12.457021 26362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:12.458621 26373 mvcc.cc:205] Tried to move back new op lower bound from 7334325831507701760 to 7334325831498289152. Current Snapshot: MvccSnapshot[applied={T|T < 7334325831507701760}]
I20260928 14:48:12.458990 26362 catalog_manager.cc:1347] Generated new cluster ID: 246efc5433c74ef98fff82321d7b48c5
I20260928 14:48:12.459041 26362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:48:12 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:12.465605 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:12.454530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.18.96.190:42625)
0928 14:48:12.454759 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:12.454769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:12.454805 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:48:12.454924 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:12.454931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:12.454989 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:12.455139 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:12.456017 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.457775 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.458714 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.459274 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.459413 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:12.461770 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:12.461773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:12.461774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:12.461786 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:12.464066 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.464449 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.464451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.464452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.464541 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.464745 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.464746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.464746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.464998 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:12.465184 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:12.465251 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 14:48:12.465389 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":117,"threads_started":1}
I20260928 14:48:12.467015 26362 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:12.467502 26362 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:12.475730 26362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7: Generated new TSK 0
I20260928 14:48:12.476105 26362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab.
Entry for principal oryx/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab.
I20260928 14:48:12.511607 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:42625
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab
--principal=oryx/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:12.600131 26377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:12.600366 26377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:12.600419 26377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:12.601641 26377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:12.601687 26377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:12.601711 26377 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:48:12.601773 26377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:12.603507 26377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab
--principal=oryx/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.190:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:12.603788 26377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:12.604058 26377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:12.609277 26385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:12.609267 26382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:12.609267 26383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:12.609473 26377 server_base.cc:1034] running on GCE node
I20260928 14:48:12.609833 26377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:12.610086 26377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:12.611248 26377 hybrid_clock.cc:648] HybridClock initialized: now 1790606892611243 us; error 33 us; skew 500 ppm
Sep 28 14:48:12 dist-test-slave-j968 krb5kdc[26260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:12.613799 26377 init.cc:377] Logged in from keytab as oryx/127.18.96.129@KRBTEST.COM (short username oryx)
I20260928 14:48:12.614221 26377 webserver.cc:460] Webserver started at http://127.18.96.129:38063/ using document root <none> and password file <none>
I20260928 14:48:12.614460 26377 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:12.614539 26377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:12.614687 26377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:12.615724 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "2e341f6d73d8425fb4dca59f99d6d569"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "5740e13e5a0029c8be09f89c87e68b50"
server_key_iv: "0ed54ceac5d085a966f876a85e68825a"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.616110 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "2e341f6d73d8425fb4dca59f99d6d569"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "5740e13e5a0029c8be09f89c87e68b50"
server_key_iv: "0ed54ceac5d085a966f876a85e68825a"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.617421 26377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:12.618170 26392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:12.618341 26377 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:48:12.618439 26377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "2e341f6d73d8425fb4dca59f99d6d569"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "5740e13e5a0029c8be09f89c87e68b50"
server_key_iv: "0ed54ceac5d085a966f876a85e68825a"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.618548 26377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:12.652076 26377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:12.653924 26377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:12.654100 26377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:12.654644 26377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:12.654711 26377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:12.654747 26377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:12.654773 26377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:48:12 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:12.662331 26377 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:32881
I20260928 14:48:12.662412 26507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:32881 every 8 connection(s)
I20260928 14:48:12.662739 26377 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:12.665419 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:12.655107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:36481 (local address 127.18.96.190:42625)
0928 14:48:12.655182 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:12.655184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:12.655456 (+ 272us) server_negotiation.cc:407] Connection header received
0928 14:48:12.656262 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:12.656263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:12.656276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:12.656333 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:12.657320 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.657856 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.659534 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.659888 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.659960 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:12.662531 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:12.662533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:12.662534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:12.662544 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:12.664087 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.664484 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.664486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.664486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.664545 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.664793 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.664793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.664794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.664906 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:12.664979 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:12.665248 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 14:48:12.665332 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:48:12.665622 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:12.655098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42625 (local address 127.18.96.129:36481)
0928 14:48:12.655295 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:12.655351 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:12.656123 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:12.656422 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:12.656426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:12.656833 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:12.657163 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:12.657170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.657931 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.657932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:12.659435 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:12.659438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.659983 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.659984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:12.660171 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:12.660685 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:12.660697 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:12.662431 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:12.664181 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:12.664188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:12.664190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:12.664407 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:12.664612 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:12.664613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:12.664613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:12.664726 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:12.664970 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:12.664977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:12.665117 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 14:48:12.665226 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:48:12.665382 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260928 14:48:12.666088 26508 heartbeater.cc:347] Connected to a master server at 127.18.96.190:42625
I20260928 14:48:12.666172 26508 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:12.666366 26508 heartbeater.cc:511] Master 127.18.96.190:42625 requested a full tablet report, sending...
I20260928 14:48:12.666946 26304 ts_manager.cc:194] Registered new tserver with Master: 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881)
I20260928 14:48:12.667598 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 26377
I20260928 14:48:12.667714 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:12.667903 26304 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:36481
I20260928 14:48:12.667959 18818 external_mini_cluster.cc:1428] Setting key 7d6acb14702a03e29423d2b6adcca17a
I20260928 14:48:12.673372 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:12.667195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:60333 (local address 127.18.96.190:42625)
0928 14:48:12.667234 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:12.667236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:12.667266 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:48:12.667336 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:12.667337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:12.667344 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:12.667388 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:12.667770 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.668531 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.669305 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.669614 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.669688 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:12.670563 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:12.670565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:12.670565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:12.670571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:12.672112 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.672525 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.672526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.672526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.672605 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.672812 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.672813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.672813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.672969 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:12.673038 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:12.673216 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 14:48:12.673287 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:48:12.673384 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:12.667037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42625 (local address 127.18.96.129:60333)
0928 14:48:12.667207 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:12.667219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:12.667280 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:12.667393 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:12.667393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:12.667575 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:12.667637 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:12.667678 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.668645 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.668646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:12.669225 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:12.669227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.669689 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.669690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:12.669737 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:12.670175 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:12.670180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:12.670458 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:12.672202 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:12.672203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:12.672203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:12.672442 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:12.672659 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:12.672659 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:12.672660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:12.672735 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:12.673041 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:12.673043 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:12.673105 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 14:48:12.673177 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:48:12.673280 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab.
Entry for principal oryx/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab.
I20260928 14:48:12.695127 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:42625
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab
--principal=oryx/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:12.791404 26517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:12.791625 26517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:12.791651 26517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:12.792856 26517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:12.792902 26517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:12.792929 26517 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:48:12.792948 26517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:12.794713 26517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab
--principal=oryx/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.190:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:12.794991 26517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:12.795244 26517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:12.800925 26523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:12.801136 26525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:12.801256 26522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:12.802347 26517 server_base.cc:1034] running on GCE node
I20260928 14:48:12.802637 26517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:12.802963 26517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:12.804133 26517 hybrid_clock.cc:648] HybridClock initialized: now 1790606892804105 us; error 48 us; skew 500 ppm
Sep 28 14:48:12 dist-test-slave-j968 krb5kdc[26260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:12.807372 26517 init.cc:377] Logged in from keytab as oryx/127.18.96.130@KRBTEST.COM (short username oryx)
I20260928 14:48:12.807921 26517 webserver.cc:460] Webserver started at http://127.18.96.130:45339/ using document root <none> and password file <none>
I20260928 14:48:12.808173 26517 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:12.808223 26517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:12.808363 26517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:12.809474 26517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "20e7132ccc2b4d08a7b2076426d0e7de"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "77e35d177f0acdb54be0841dfe969667"
server_key_iv: "f0a5fd656417ea4cb57a4b3f103a581b"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.809866 26517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "20e7132ccc2b4d08a7b2076426d0e7de"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "77e35d177f0acdb54be0841dfe969667"
server_key_iv: "f0a5fd656417ea4cb57a4b3f103a581b"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.811399 26517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:12.812305 26532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:12.812532 26517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:48:12.812618 26517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "20e7132ccc2b4d08a7b2076426d0e7de"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "77e35d177f0acdb54be0841dfe969667"
server_key_iv: "f0a5fd656417ea4cb57a4b3f103a581b"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.812677 26517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:12.822216 26517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:12.824435 26517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:12.824558 26517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:12.825091 26517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:12.825139 26517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:12.825162 26517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:12.825176 26517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:12.832419 26517 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:39197
I20260928 14:48:12.832561 26647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:39197 every 8 connection(s)
Sep 28 14:48:12 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:12.832880 26517 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:12.836263 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:12.825592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:52363 (local address 127.18.96.190:42625)
0928 14:48:12.825642 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:12.825646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:12.825938 (+ 292us) server_negotiation.cc:407] Connection header received
0928 14:48:12.826457 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:12.826458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:12.826513 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:12.826594 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:12.827574 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.828373 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.830026 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.830339 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.830411 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:12.833314 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:12.833316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:12.833316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:12.833325 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:12.834791 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.835220 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.835221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.835222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.835272 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.835468 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.835469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.835469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.835571 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:12.835632 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:12.836083 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 14:48:12.836170 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:48:12.836481 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:12.825624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42625 (local address 127.18.96.130:52363)
0928 14:48:12.825793 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:12.825836 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:12.826348 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:12.826735 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:12.826739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:12.827174 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:12.827472 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:12.827478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.828469 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.828470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:12.829936 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:12.829940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.830418 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.830419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:12.830588 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:12.831178 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:12.831191 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:12.833217 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:12.834897 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:12.834902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:12.834907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:12.835139 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:12.835314 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:12.835315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:12.835315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:12.835399 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:12.835661 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:12.835667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:12.835787 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 14:48:12.836047 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 14:48:12.836217 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260928 14:48:12.836951 26648 heartbeater.cc:347] Connected to a master server at 127.18.96.190:42625
I20260928 14:48:12.837044 26648 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:12.837239 26648 heartbeater.cc:511] Master 127.18.96.190:42625 requested a full tablet report, sending...
I20260928 14:48:12.837575 26305 ts_manager.cc:194] Registered new tserver with Master: 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130:39197)
I20260928 14:48:12.837967 26305 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:52363
I20260928 14:48:12.840838 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 26517
I20260928 14:48:12.841041 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:12.841288 18818 external_mini_cluster.cc:1428] Setting key 5dc9773d5520e79f61caae37d4bcbc4d
I20260928 14:48:12.844224 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:12.838069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42625 (local address 127.18.96.130:42965)
0928 14:48:12.838205 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:12.838216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:12.838276 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:12.838425 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:12.838425 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:12.838577 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:12.838636 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:12.838637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.839304 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.839305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:12.839811 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:12.839812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.840267 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.840268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:12.840321 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:12.840812 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:12.840817 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:12.841239 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:12.842940 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:12.842940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:12.842941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:12.843125 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:12.843360 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:12.843361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:12.843362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:12.843452 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:12.843908 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:12.843909 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:12.843970 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 14:48:12.844048 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:48:12.844145 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260928 14:48:12.844281 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:12.838110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:42965 (local address 127.18.96.190:42625)
0928 14:48:12.838154 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:12.838156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:12.838329 (+ 173us) server_negotiation.cc:407] Connection header received
0928 14:48:12.838350 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:12.838351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:12.838361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:12.838420 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:12.838709 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.839222 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.839949 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:12.840195 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:12.840265 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:12.841401 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:12.841403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:12.841403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:12.841410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:12.842855 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.843209 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.843211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.843212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.843270 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:12.843544 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:12.843545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:12.843546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:12.843763 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:12.843886 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:12.844095 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 14:48:12.844185 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for oryx/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab.
Entry for principal oryx/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab.
I20260928 14:48:12.870779 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:42625
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab
--principal=oryx/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:12.953670 26657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:12.953898 26657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:12.953925 26657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:12.955077 26657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:12.955127 26657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:12.955147 26657 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:48:12.955165 26657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:12.957026 26657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab
--principal=oryx/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.190:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:12.957239 26657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:12.957480 26657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:12.962910 26662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:12.962877 26665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:12.962875 26663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:12.963109 26657 server_base.cc:1034] running on GCE node
I20260928 14:48:12.963323 26657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:12.963631 26657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:12.964778 26657 hybrid_clock.cc:648] HybridClock initialized: now 1790606892964751 us; error 51 us; skew 500 ppm
Sep 28 14:48:12 dist-test-slave-j968 krb5kdc[26260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:12.967543 26657 init.cc:377] Logged in from keytab as oryx/127.18.96.131@KRBTEST.COM (short username oryx)
I20260928 14:48:12.968084 26657 webserver.cc:460] Webserver started at http://127.18.96.131:36911/ using document root <none> and password file <none>
I20260928 14:48:12.968281 26657 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:12.968320 26657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:12.968410 26657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:12.969451 26657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "67d03ab2e91b4c5cb38a7b3da6d95505"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "4661ae7527df6eb75e2805e8990358d5"
server_key_iv: "23b985bb11ad59c0970a6b75f0d4b6ce"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.969758 26657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "67d03ab2e91b4c5cb38a7b3da6d95505"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "4661ae7527df6eb75e2805e8990358d5"
server_key_iv: "23b985bb11ad59c0970a6b75f0d4b6ce"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.971148 26657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:12.971973 26672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:12.972174 26657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:12.972276 26657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "67d03ab2e91b4c5cb38a7b3da6d95505"
format_stamp: "Formatted at 2026-09-28 14:48:12 on dist-test-slave-j968"
server_key: "4661ae7527df6eb75e2805e8990358d5"
server_key_iv: "23b985bb11ad59c0970a6b75f0d4b6ce"
server_key_version: "encryptionkey@0"
I20260928 14:48:12.972361 26657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:13.000494 26657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:13.002663 26657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:13.002842 26657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:13.003500 26657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:13.003552 26657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.003600 26657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:13.003616 26657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:13.011381 26657 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:41641
I20260928 14:48:13.011590 26787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:41641 every 8 connection(s)
I20260928 14:48:13.011798 26657 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:13.014497 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.003961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:54243 (local address 127.18.96.190:42625)
0928 14:48:13.004028 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.004031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.004371 (+ 340us) server_negotiation.cc:407] Connection header received
0928 14:48:13.005217 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.005218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.005233 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.005314 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:13.006279 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.006829 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.008167 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.008411 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.008506 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.011317 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:13.011319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:13.011319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:13.011328 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:13.013217 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.013584 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.013585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.013585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.013641 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.013836 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.013837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.013837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.014027 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:13.014088 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.014336 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.014420 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:48:13.014576 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.004032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42625 (local address 127.18.96.131:54243)
0928 14:48:13.004214 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.004266 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.005022 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.005431 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.005436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.005811 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:13.006159 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.006167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.006986 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.006987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.008034 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.008037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.008532 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.008534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.008713 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.009335 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:13.009346 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:13.011200 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:13.013290 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:13.013294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:13.013296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:13.013507 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:13.013693 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:13.013693 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:13.013694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:13.013781 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:13.014097 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:13.014102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:13.014215 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.014299 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.014417 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260928 14:48:13.015159 26788 heartbeater.cc:347] Connected to a master server at 127.18.96.190:42625
I20260928 14:48:13.015280 26788 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:13.015471 26788 heartbeater.cc:511] Master 127.18.96.190:42625 requested a full tablet report, sending...
I20260928 14:48:13.016032 26304 ts_manager.cc:194] Registered new tserver with Master: 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641)
I20260928 14:48:13.016219 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 26657
I20260928 14:48:13.016345 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:13.016573 18818 external_mini_cluster.cc:1428] Setting key 6c4b845f0df5449d74022fc2b32972ff
I20260928 14:48:13.016575 26304 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:54243
I20260928 14:48:13.017490 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:48:13.023293 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.016170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42625 (local address 127.18.96.131:37165)
0928 14:48:13.016373 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.016387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.016442 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.017014 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.017015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.017272 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:13.017352 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.017354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.018273 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.018274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.018865 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.018867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.019352 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.019353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.019469 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.019916 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:13.019922 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:13.020397 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:13.022144 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:13.022145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:13.022146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:13.022399 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:13.022602 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:13.022603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:13.022603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:13.022676 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:13.022901 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:13.022903 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:13.022958 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.023073 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.023195 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260928 14:48:13.023336 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.016735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:37165 (local address 127.18.96.190:42625)
0928 14:48:13.016793 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.016795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.016804 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:13.016824 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.016825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.016836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.016895 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:13.017471 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.018167 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.018945 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.019264 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.019338 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.020505 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:13.020506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:13.020507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:13.020513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:13.022069 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.022485 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.022486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.022486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.022534 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.022738 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.022739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.022739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.022832 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:13.022883 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.023133 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.023217 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606893, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:48:13.039987 26355 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:13.034433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.18.96.190:42625)
0928 14:48:13.034501 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.034503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.034512 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:13.034544 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.034545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.034555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.034610 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:13.035157 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.035644 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.036449 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.036644 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.036696 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.039790 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42106: BlockingRecv error: recv got EOF from 127.0.0.1:42106 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606893, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:13.047201 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.040773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.18.96.190:42625)
0928 14:48:13.040825 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.040827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.040861 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:48:13.040915 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.040916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.040924 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.040964 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:13.041427 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.041911 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.042635 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.042811 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.042862 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.045010 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:13.045011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:13.045011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:13.045018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:13.046290 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.046592 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.046593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.046593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.046644 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.046809 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.046809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.046809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.046910 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:13.047022 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.047071 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.047133 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:48:13.049180 26304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:13.050787 26304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:13.057569 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.053563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.18.96.131:41641)
0928 14:48:13.053728 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.053734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.053743 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:13.053783 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.053785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.053802 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.053886 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:13.054157 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.054981 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.056516 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.057152 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.057242 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.057324 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.057439 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.057476 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260928 14:48:13.057564 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.053287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:41641 (local address 127.0.0.1:36326)
0928 14:48:13.053581 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.053591 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.053665 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.053992 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.053996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.053998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:13.054074 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.054075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.055063 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.055064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.056432 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.056434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.057244 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.057245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.057334 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.057350 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.057424 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.057476 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260928 14:48:13.057726 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.053222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:32881 (local address 127.0.0.1:37000)
0928 14:48:13.053477 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.053517 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.053595 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.054061 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.054062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.054064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:13.054117 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.054119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.055473 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.055474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.056589 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.056591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.057476 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.057477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.057544 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.057552 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.057583 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.057626 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":5,"thread_start_us":69,"threads_started":1}
I20260928 14:48:13.058151 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.053287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39197 (local address 127.0.0.1:47548)
0928 14:48:13.053713 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.053721 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.053760 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.054191 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.054192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.054194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:13.054222 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.054223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.055504 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.055505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.056526 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.056529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.057476 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.057477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.057544 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.057551 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.057583 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.058082 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":5,"thread_start_us":132,"threads_started":1}
I20260928 14:48:13.058167 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.053325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.18.96.129:32881)
0928 14:48:13.053880 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.053885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.053895 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:13.053935 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.053937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.053953 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.054193 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:13.054258 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.055373 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.056673 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.057376 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.057922 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.058008 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.058033 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.058076 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":93,"threads_started":1}
I20260928 14:48:13.058231 26722 tablet_service.cc:1467] Processing CreateTablet for tablet 27d7c51c053a43bd8265edd7f93b82b9 (DEFAULT_TABLE table=test-table [id=dbb0232dfe644f5c9ad6c46d64151ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.058641 26722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d7c51c053a43bd8265edd7f93b82b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.058805 26431 tablet_service.cc:1467] Processing CreateTablet for tablet 27d7c51c053a43bd8265edd7f93b82b9 (DEFAULT_TABLE table=test-table [id=dbb0232dfe644f5c9ad6c46d64151ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.059155 26431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d7c51c053a43bd8265edd7f93b82b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.061791 26816 tablet_bootstrap.cc:492] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: Bootstrap starting.
I20260928 14:48:13.062636 26816 tablet_bootstrap.cc:654] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.062659 26817 tablet_bootstrap.cc:492] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: Bootstrap starting.
I20260928 14:48:13.062952 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.053762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.18.96.130:39197)
0928 14:48:13.054020 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.054024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.054034 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:13.054068 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.054070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.054092 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.054291 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:13.054350 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.055406 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.056616 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.057357 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.058333 (+ 976us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.062748 (+ 4415us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.062785 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.062841 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":78,"threads_started":1}
I20260928 14:48:13.062974 26816 log.cc:824] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:13.063305 26817 tablet_bootstrap.cc:654] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.063583 26582 tablet_service.cc:1467] Processing CreateTablet for tablet 27d7c51c053a43bd8265edd7f93b82b9 (DEFAULT_TABLE table=test-table [id=dbb0232dfe644f5c9ad6c46d64151ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.063650 26817 log.cc:824] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:13.063748 26816 tablet_bootstrap.cc:492] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: No bootstrap required, opened a new log
I20260928 14:48:13.063810 26816 ts_tablet_manager.cc:1397] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:13.063920 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d7c51c053a43bd8265edd7f93b82b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.064471 26817 tablet_bootstrap.cc:492] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: No bootstrap required, opened a new log
I20260928 14:48:13.064615 26817 ts_tablet_manager.cc:1397] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:13.065881 26816 raft_consensus.cc:348] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } }
I20260928 14:48:13.066056 26816 raft_consensus.cc:374] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.066103 26816 raft_consensus.cc:729] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d03ab2e91b4c5cb38a7b3da6d95505, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.066241 26816 consensus_queue.cc:260] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } }
I20260928 14:48:13.066502 26816 ts_tablet_manager.cc:1428] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:13.066619 26788 heartbeater.cc:503] Master 127.18.96.190:42625 was elected leader, sending a full tablet report...
I20260928 14:48:13.066677 26817 raft_consensus.cc:348] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } }
I20260928 14:48:13.066823 26817 raft_consensus.cc:374] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.066869 26817 raft_consensus.cc:729] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e341f6d73d8425fb4dca59f99d6d569, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.066998 26817 consensus_queue.cc:260] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } }
I20260928 14:48:13.067250 26817 ts_tablet_manager.cc:1428] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:48:13.067281 26508 heartbeater.cc:503] Master 127.18.96.190:42625 was elected leader, sending a full tablet report...
I20260928 14:48:13.067652 26822 tablet_bootstrap.cc:492] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: Bootstrap starting.
I20260928 14:48:13.068305 26822 tablet_bootstrap.cc:654] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.068559 26822 log.cc:824] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:13.069300 26822 tablet_bootstrap.cc:492] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: No bootstrap required, opened a new log
I20260928 14:48:13.069365 26822 ts_tablet_manager.cc:1397] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:48:13.070667 26822 raft_consensus.cc:348] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } }
I20260928 14:48:13.070780 26822 raft_consensus.cc:374] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.070801 26822 raft_consensus.cc:729] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e7132ccc2b4d08a7b2076426d0e7de, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.070904 26822 consensus_queue.cc:260] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } }
I20260928 14:48:13.071163 26822 ts_tablet_manager.cc:1428] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:13.071240 26648 heartbeater.cc:503] Master 127.18.96.190:42625 was elected leader, sending a full tablet report...
W20260928 14:48:13.083455 26649 tablet.cc:2369] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:48:13.163808 26509 tablet.cc:2369] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:13.257834 26824 raft_consensus.cc:482] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:13.257969 26824 raft_consensus.cc:504] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } }
I20260928 14:48:13.258317 26824 leader_election.cc:288] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881), 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641)
I20260928 14:48:13.262365 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.258577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:44287 (local address 127.18.96.129:32881)
0928 14:48:13.258637 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.258640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.258884 (+ 244us) server_negotiation.cc:407] Connection header received
0928 14:48:13.258913 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.258915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.259000 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.259061 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:13.259212 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.259929 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.261346 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.262034 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.262108 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.262122 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.262247 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.262287 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:48:13.262374 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.258618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:41293 (local address 127.18.96.131:41641)
0928 14:48:13.258659 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.258660 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.258672 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:48:13.258767 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.258768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.258775 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.258828 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:13.259033 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.259832 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.261346 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.261957 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.262046 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.262056 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.262247 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.262287 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:48:13.262401 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.258508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:41641 (local address 127.18.96.130:41293)
0928 14:48:13.258585 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.258605 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.258690 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.258863 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.258864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.258866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:13.258949 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.258950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.259946 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.259947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.261221 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.261225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.262048 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.262049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.262154 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.262158 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.262221 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.262274 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":7}
I20260928 14:48:13.262401 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.258519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:32881 (local address 127.18.96.130:44287)
0928 14:48:13.258801 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.258812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.258862 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.259088 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.259089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.259090 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:13.259141 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.259142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.260025 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.260026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.261221 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.261225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.262110 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.262111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.262168 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.262171 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.262223 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.262274 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":123,"thread_start_us":132,"threads_started":1}
W20260928 14:48:13.262637 26789 tablet.cc:2369] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:13.262820 26462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d7c51c053a43bd8265edd7f93b82b9" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e341f6d73d8425fb4dca59f99d6d569" is_pre_election: true
I20260928 14:48:13.262802 26742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d7c51c053a43bd8265edd7f93b82b9" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" is_pre_election: true
I20260928 14:48:13.262990 26462 raft_consensus.cc:2454] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 0.
I20260928 14:48:13.263032 26742 raft_consensus.cc:2454] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 0.
I20260928 14:48:13.263243 26534 leader_election.cc:302] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e7132ccc2b4d08a7b2076426d0e7de, 2e341f6d73d8425fb4dca59f99d6d569; no voters:
I20260928 14:48:13.263386 26824 raft_consensus.cc:2781] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:13.263444 26824 raft_consensus.cc:482] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:13.263473 26824 raft_consensus.cc:3037] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.264496 26824 raft_consensus.cc:504] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } }
I20260928 14:48:13.264658 26824 leader_election.cc:288] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 election: Requested vote from peers 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881), 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641)
I20260928 14:48:13.264806 26462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d7c51c053a43bd8265edd7f93b82b9" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e341f6d73d8425fb4dca59f99d6d569"
I20260928 14:48:13.264890 26462 raft_consensus.cc:3037] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.264883 26742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d7c51c053a43bd8265edd7f93b82b9" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505"
I20260928 14:48:13.265055 26742 raft_consensus.cc:3037] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.265635 26462 raft_consensus.cc:2454] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 1.
I20260928 14:48:13.265794 26742 raft_consensus.cc:2454] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 1.
I20260928 14:48:13.265800 26534 leader_election.cc:302] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e7132ccc2b4d08a7b2076426d0e7de, 2e341f6d73d8425fb4dca59f99d6d569; no voters:
I20260928 14:48:13.265975 26824 raft_consensus.cc:2781] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:13.266166 26824 raft_consensus.cc:686] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 LEADER]: Becoming Leader. State: Replica: 20e7132ccc2b4d08a7b2076426d0e7de, State: Running, Role: LEADER
I20260928 14:48:13.266271 26824 consensus_queue.cc:237] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } }
I20260928 14:48:13.267320 26305 catalog_manager.cc:5495] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130). New cstate: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } health_report { overall_health: HEALTHY } } }
I20260928 14:48:13.283792 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.281273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.18.96.130:39197)
0928 14:48:13.281347 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.281348 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.281358 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:13.281392 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.281393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.281410 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.281481 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:13.281609 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.282128 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.283027 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.283237 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.283308 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.283476 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.283589 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.283660 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.283701 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.283723 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:48:13.285964 26742 raft_consensus.cc:1261] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Refusing update from remote peer 20e7132ccc2b4d08a7b2076426d0e7de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.285949 26462 raft_consensus.cc:1261] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Refusing update from remote peer 20e7132ccc2b4d08a7b2076426d0e7de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.286348 26826 consensus_queue.cc:1035] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.286513 26824 consensus_queue.cc:1035] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.289361 26830 mvcc.cc:205] Tried to move back new op lower bound from 7334325834896199680 to 7334325834819231744. Current Snapshot: MvccSnapshot[applied={T|T < 7334325834896199680}]
I20260928 14:48:13.289614 26831 mvcc.cc:205] Tried to move back new op lower bound from 7334325834896199680 to 7334325834819231744. Current Snapshot: MvccSnapshot[applied={T|T < 7334325834896199680}]
I20260928 14:48:13.291620 26305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:13.291715 26305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:13.292538 26305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7: Sending DeleteTablet for 3 replicas of tablet 27d7c51c053a43bd8265edd7f93b82b9
I20260928 14:48:13.292882 26582 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d7c51c053a43bd8265edd7f93b82b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:47548
I20260928 14:48:13.292904 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d7c51c053a43bd8265edd7f93b82b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:36326
I20260928 14:48:13.293136 26839 tablet_replica.cc:330] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: stopping tablet replica
I20260928 14:48:13.293154 26431 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d7c51c053a43bd8265edd7f93b82b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:37000
I20260928 14:48:13.293242 26839 raft_consensus.cc:2229] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:48:13.293387 26839 raft_consensus.cc:2258] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.293618 26305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:13.293774 26305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:13.293831 26840 tablet_replica.cc:330] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: stopping tablet replica
I20260928 14:48:13.293892 26839 ts_tablet_manager.cc:1905] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.293916 26840 raft_consensus.cc:2229] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:13.294031 26840 raft_consensus.cc:2258] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.294178 26841 tablet_replica.cc:330] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: stopping tablet replica
I20260928 14:48:13.294262 26841 raft_consensus.cc:2229] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:13.294341 26841 raft_consensus.cc:2258] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.294792 26841 ts_tablet_manager.cc:1905] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.295433 26722 tablet_service.cc:1467] Processing CreateTablet for tablet 8160815e9bb942278ae360db54e908b5 (DEFAULT_TABLE table=test-table [id=e6fb92d01f0142b6b344d95a79596f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.295558 26722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8160815e9bb942278ae360db54e908b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.295745 26839 ts_tablet_manager.cc:1918] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:13.295809 26839 log.cc:1196] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/27d7c51c053a43bd8265edd7f93b82b9
I20260928 14:48:13.296092 26839 ts_tablet_manager.cc:1939] T 27d7c51c053a43bd8265edd7f93b82b9 P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting consensus metadata
I20260928 14:48:13.296505 26841 ts_tablet_manager.cc:1918] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:13.296558 26841 log.cc:1196] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/27d7c51c053a43bd8265edd7f93b82b9
I20260928 14:48:13.296782 26841 ts_tablet_manager.cc:1939] T 27d7c51c053a43bd8265edd7f93b82b9 P 2e341f6d73d8425fb4dca59f99d6d569: Deleting consensus metadata
I20260928 14:48:13.296927 26816 tablet_bootstrap.cc:492] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: Bootstrap starting.
I20260928 14:48:13.297178 26431 tablet_service.cc:1467] Processing CreateTablet for tablet 8160815e9bb942278ae360db54e908b5 (DEFAULT_TABLE table=test-table [id=e6fb92d01f0142b6b344d95a79596f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.297277 26431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8160815e9bb942278ae360db54e908b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.297308 26290 catalog_manager.cc:4841] TS 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881): tablet 27d7c51c053a43bd8265edd7f93b82b9 (table test-table [id=dbb0232dfe644f5c9ad6c46d64151ae2]) successfully deleted
I20260928 14:48:13.297350 26816 tablet_bootstrap.cc:654] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.297921 26840 ts_tablet_manager.cc:1905] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.298468 26817 tablet_bootstrap.cc:492] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: Bootstrap starting.
I20260928 14:48:13.298566 26291 catalog_manager.cc:4841] TS 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130:39197): tablet 27d7c51c053a43bd8265edd7f93b82b9 (table test-table [id=dbb0232dfe644f5c9ad6c46d64151ae2]) successfully deleted
I20260928 14:48:13.298838 26817 tablet_bootstrap.cc:654] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.299197 26840 ts_tablet_manager.cc:1918] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:13.299252 26840 log.cc:1196] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/27d7c51c053a43bd8265edd7f93b82b9
I20260928 14:48:13.299494 26840 ts_tablet_manager.cc:1939] T 27d7c51c053a43bd8265edd7f93b82b9 P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting consensus metadata
I20260928 14:48:13.299608 26817 tablet_bootstrap.cc:492] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: No bootstrap required, opened a new log
I20260928 14:48:13.299646 26817 ts_tablet_manager.cc:1397] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:13.299736 26582 tablet_service.cc:1467] Processing CreateTablet for tablet 8160815e9bb942278ae360db54e908b5 (DEFAULT_TABLE table=test-table [id=e6fb92d01f0142b6b344d95a79596f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.299824 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8160815e9bb942278ae360db54e908b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.299785 26817 raft_consensus.cc:348] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.299855 26817 raft_consensus.cc:374] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.299902 26817 raft_consensus.cc:729] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e341f6d73d8425fb4dca59f99d6d569, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.299949 26817 consensus_queue.cc:260] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.299997 26289 catalog_manager.cc:4841] TS 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641): tablet 27d7c51c053a43bd8265edd7f93b82b9 (table test-table [id=dbb0232dfe644f5c9ad6c46d64151ae2]) successfully deleted
I20260928 14:48:13.300067 26817 ts_tablet_manager.cc:1428] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.301198 26822 tablet_bootstrap.cc:492] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: Bootstrap starting.
I20260928 14:48:13.301355 26816 tablet_bootstrap.cc:492] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: No bootstrap required, opened a new log
I20260928 14:48:13.301398 26816 ts_tablet_manager.cc:1397] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 14:48:13.301561 26816 raft_consensus.cc:348] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.301637 26816 raft_consensus.cc:374] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.301640 26822 tablet_bootstrap.cc:654] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.301661 26816 raft_consensus.cc:729] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d03ab2e91b4c5cb38a7b3da6d95505, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.301716 26816 consensus_queue.cc:260] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.301831 26816 ts_tablet_manager.cc:1428] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.302345 26822 tablet_bootstrap.cc:492] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: No bootstrap required, opened a new log
I20260928 14:48:13.302400 26822 ts_tablet_manager.cc:1397] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:13.302560 26822 raft_consensus.cc:348] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.302654 26822 raft_consensus.cc:374] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.302678 26822 raft_consensus.cc:729] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e7132ccc2b4d08a7b2076426d0e7de, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.302726 26822 consensus_queue.cc:260] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.302804 26822 ts_tablet_manager.cc:1428] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.423801 26834 raft_consensus.cc:482] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:13.423947 26834 raft_consensus.cc:504] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.424175 26834 leader_election.cc:288] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881), 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641)
I20260928 14:48:13.424403 26742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8160815e9bb942278ae360db54e908b5" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" is_pre_election: true
I20260928 14:48:13.424516 26742 raft_consensus.cc:2454] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 0.
I20260928 14:48:13.424682 26462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8160815e9bb942278ae360db54e908b5" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e341f6d73d8425fb4dca59f99d6d569" is_pre_election: true
I20260928 14:48:13.424711 26533 leader_election.cc:302] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e7132ccc2b4d08a7b2076426d0e7de, 67d03ab2e91b4c5cb38a7b3da6d95505; no voters:
I20260928 14:48:13.424791 26462 raft_consensus.cc:2454] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 0.
I20260928 14:48:13.424837 26834 raft_consensus.cc:2781] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:13.424880 26834 raft_consensus.cc:482] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:13.424909 26834 raft_consensus.cc:3037] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.425661 26834 raft_consensus.cc:504] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.425814 26834 leader_election.cc:288] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 election: Requested vote from peers 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881), 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641)
I20260928 14:48:13.425997 26742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8160815e9bb942278ae360db54e908b5" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505"
I20260928 14:48:13.426084 26742 raft_consensus.cc:3037] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.426123 26462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8160815e9bb942278ae360db54e908b5" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e341f6d73d8425fb4dca59f99d6d569"
I20260928 14:48:13.426213 26462 raft_consensus.cc:3037] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.426718 26742 raft_consensus.cc:2454] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 1.
I20260928 14:48:13.426923 26533 leader_election.cc:302] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e7132ccc2b4d08a7b2076426d0e7de, 67d03ab2e91b4c5cb38a7b3da6d95505; no voters:
I20260928 14:48:13.427060 26462 raft_consensus.cc:2454] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 1.
I20260928 14:48:13.427075 26834 raft_consensus.cc:2781] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:13.427134 26834 raft_consensus.cc:686] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 LEADER]: Becoming Leader. State: Replica: 20e7132ccc2b4d08a7b2076426d0e7de, State: Running, Role: LEADER
I20260928 14:48:13.427191 26834 consensus_queue.cc:237] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.427932 26304 catalog_manager.cc:5495] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130). New cstate: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606892, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.190@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:13.472049 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.464671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.18.96.190:42625)
0928 14:48:13.464755 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.464757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.464885 (+ 128us) server_negotiation.cc:407] Connection header received
0928 14:48:13.465012 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.465013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.465023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.465091 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:13.465667 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.466167 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.466984 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.467162 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.467252 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.469390 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:13.469392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:13.469392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:13.469399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:13.470708 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.471144 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.471145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.471146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.471203 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.471412 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.471413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.471413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.471523 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:13.471599 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.471753 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.471950 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":36}
I20260928 14:48:13.472049 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.464598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42625 (local address 127.0.0.1:42118)
0928 14:48:13.464810 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.464820 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.464904 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.465094 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.465095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.465505 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:13.465567 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.465570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.466253 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.466254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.466888 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.466890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.467262 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.467263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.467321 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.467823 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:13.467827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:13.469287 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:13.470818 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:13.470826 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:13.470829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:13.471050 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:13.471270 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:13.471271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:13.471271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:13.471331 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:13.471589 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:13.471596 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:13.471639 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.471717 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.471875 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":131,"threads_started":1}
I20260928 14:48:13.474046 26304 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:42118:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:48:13.474306 26304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:13.475697 26431 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5e464f5eb04ecfa17d031e33c3d791 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec7e68e391d8493cad3b935d1faaba0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:48:13.475785 26722 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5e464f5eb04ecfa17d031e33c3d791 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec7e68e391d8493cad3b935d1faaba0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:48:13.475889 26431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5e464f5eb04ecfa17d031e33c3d791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.475922 26722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5e464f5eb04ecfa17d031e33c3d791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.477432 26816 tablet_bootstrap.cc:492] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: Bootstrap starting.
I20260928 14:48:13.477576 26582 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5e464f5eb04ecfa17d031e33c3d791 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec7e68e391d8493cad3b935d1faaba0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:48:13.477735 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5e464f5eb04ecfa17d031e33c3d791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.477901 26816 tablet_bootstrap.cc:654] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.478253 26817 tablet_bootstrap.cc:492] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: Bootstrap starting.
I20260928 14:48:13.478631 26816 tablet_bootstrap.cc:492] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: No bootstrap required, opened a new log
I20260928 14:48:13.478684 26817 tablet_bootstrap.cc:654] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.478682 26816 ts_tablet_manager.cc:1397] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:13.478842 26816 raft_consensus.cc:348] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.478910 26822 tablet_bootstrap.cc:492] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: Bootstrap starting.
I20260928 14:48:13.478925 26816 raft_consensus.cc:374] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.478948 26816 raft_consensus.cc:729] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d03ab2e91b4c5cb38a7b3da6d95505, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.478996 26816 consensus_queue.cc:260] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.479100 26816 ts_tablet_manager.cc:1428] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.479166 26820 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.479240 26822 tablet_bootstrap.cc:654] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.479257 26820 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:48:13.479722 26817 tablet_bootstrap.cc:492] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: No bootstrap required, opened a new log
I20260928 14:48:13.479775 26817 ts_tablet_manager.cc:1397] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:48:13.479857 26822 tablet_bootstrap.cc:492] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: No bootstrap required, opened a new log
I20260928 14:48:13.479926 26822 ts_tablet_manager.cc:1397] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:13.479921 26817 raft_consensus.cc:348] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.479990 26817 raft_consensus.cc:374] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.480018 26817 raft_consensus.cc:729] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e341f6d73d8425fb4dca59f99d6d569, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.480046 26822 raft_consensus.cc:348] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.480067 26817 consensus_queue.cc:260] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.480115 26822 raft_consensus.cc:374] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.480154 26822 raft_consensus.cc:729] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e7132ccc2b4d08a7b2076426d0e7de, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.480159 26817 ts_tablet_manager.cc:1428] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.480206 26822 consensus_queue.cc:260] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.480211 26821 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.480293 26822 ts_tablet_manager.cc:1428] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.480291 26821 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:48:13.480341 26834 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.480422 26834 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:48:13.669224 26834 raft_consensus.cc:482] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:13.669319 26834 raft_consensus.cc:504] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.669559 26834 leader_election.cc:288] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881), 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641)
I20260928 14:48:13.669848 26742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e464f5eb04ecfa17d031e33c3d791" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" is_pre_election: true
I20260928 14:48:13.669962 26742 raft_consensus.cc:2454] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 0.
I20260928 14:48:13.669848 26462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e464f5eb04ecfa17d031e33c3d791" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e341f6d73d8425fb4dca59f99d6d569" is_pre_election: true
I20260928 14:48:13.670032 26462 raft_consensus.cc:2454] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 0.
I20260928 14:48:13.670157 26533 leader_election.cc:302] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e7132ccc2b4d08a7b2076426d0e7de, 67d03ab2e91b4c5cb38a7b3da6d95505; no voters:
I20260928 14:48:13.670252 26834 raft_consensus.cc:2781] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:13.670291 26834 raft_consensus.cc:482] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:13.670315 26834 raft_consensus.cc:3037] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.670975 26834 raft_consensus.cc:504] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.671119 26834 leader_election.cc:288] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 election: Requested vote from peers 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881), 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641)
I20260928 14:48:13.671290 26462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e464f5eb04ecfa17d031e33c3d791" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e341f6d73d8425fb4dca59f99d6d569"
I20260928 14:48:13.671286 26742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5e464f5eb04ecfa17d031e33c3d791" candidate_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505"
I20260928 14:48:13.671375 26742 raft_consensus.cc:3037] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.671404 26462 raft_consensus.cc:3037] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.672209 26462 raft_consensus.cc:2454] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 1.
I20260928 14:48:13.672222 26742 raft_consensus.cc:2454] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e7132ccc2b4d08a7b2076426d0e7de in term 1.
I20260928 14:48:13.672479 26534 leader_election.cc:302] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e7132ccc2b4d08a7b2076426d0e7de, 2e341f6d73d8425fb4dca59f99d6d569; no voters:
I20260928 14:48:13.672610 26834 raft_consensus.cc:2781] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:13.672667 26834 raft_consensus.cc:686] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 LEADER]: Becoming Leader. State: Replica: 20e7132ccc2b4d08a7b2076426d0e7de, State: Running, Role: LEADER
I20260928 14:48:13.672704 26834 consensus_queue.cc:237] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.672968 26824 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: TxnStatusTablet state changed. Reason: New leader 20e7132ccc2b4d08a7b2076426d0e7de. Latest consensus state: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.673058 26824 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:48:13.673206 26856 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:48:13.673327 26856 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:48:13.673326 26305 catalog_manager.cc:5495] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de reported cstate change: term changed from 0 to 1, leader changed from <none> to 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130). New cstate: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } health_report { overall_health: UNKNOWN } } }
I20260928 14:48:13.707973 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.704957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39197 (local address 127.0.0.1:47560)
0928 14:48:13.705068 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.705082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.705136 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.705295 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.705296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.705297 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:48:13.705355 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.705357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.706107 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.706108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.707056 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.707059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.707466 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.707467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.707532 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.707550 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.707738 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.707747 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.707803 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.707880 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 14:48:13.707999 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.705076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.18.96.130:39197)
0928 14:48:13.705149 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.705151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.705162 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:13.705202 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.705203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.705213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.705265 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:13.705431 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.706010 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.707166 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.707352 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.707439 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.707643 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.707670 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.707715 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.707849 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.707895 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:48:13.709070 26462 raft_consensus.cc:1261] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Refusing update from remote peer 20e7132ccc2b4d08a7b2076426d0e7de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.709132 26742 raft_consensus.cc:1261] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Refusing update from remote peer 20e7132ccc2b4d08a7b2076426d0e7de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.709295 26824 consensus_queue.cc:1035] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.709389 26834 consensus_queue.cc:1035] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.709890 26820 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: TxnStatusTablet state changed. Reason: New leader 20e7132ccc2b4d08a7b2076426d0e7de. Latest consensus state: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.709882 26821 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: TxnStatusTablet state changed. Reason: New leader 20e7132ccc2b4d08a7b2076426d0e7de. Latest consensus state: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.709976 26820 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:48:13.709975 26821 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:48:13.710851 26829 mvcc.cc:205] Tried to move back new op lower bound from 7334325836630425600 to 7334325836483768320. Current Snapshot: MvccSnapshot[applied={T|T < 7334325836630425600}]
I20260928 14:48:13.710850 26826 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.711030 26826 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:48:13.711552 26821 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.711637 26821 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 2e341f6d73d8425fb4dca59f99d6d569: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:48:13.712193 26826 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.712268 26826 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 20e7132ccc2b4d08a7b2076426d0e7de: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:48:13.713228 26820 tablet_replica.cc:439] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } } }
I20260928 14:48:13.713346 26820 tablet_replica.cc:442] T 3c5e464f5eb04ecfa17d031e33c3d791 P 67d03ab2e91b4c5cb38a7b3da6d95505: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:48:13.717955 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.715479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:57929 (local address 127.18.96.130:39197)
0928 14:48:13.715518 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.715521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.715684 (+ 163us) server_negotiation.cc:407] Connection header received
0928 14:48:13.715749 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.715750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.715759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.715813 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:13.715959 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.716467 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.717245 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.717437 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.717518 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.717688 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.717725 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.717784 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.717841 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.717876 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":20}
I20260928 14:48:13.717962 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.715405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39197 (local address 127.18.96.130:57929)
0928 14:48:13.715607 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.715619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.715699 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.715815 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.715816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.715818 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:48:13.715904 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.715905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.716551 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.716552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.717149 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.717151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.717518 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.717519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.717576 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.717590 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.717783 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.717784 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.717837 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.717893 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260928 14:48:13.720296 26742 raft_consensus.cc:1261] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Refusing update from remote peer 20e7132ccc2b4d08a7b2076426d0e7de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.720461 26462 raft_consensus.cc:1261] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Refusing update from remote peer 20e7132ccc2b4d08a7b2076426d0e7de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.720606 26864 consensus_queue.cc:1035] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.720710 26834 consensus_queue.cc:1035] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.732595 26304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:13.732730 26304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:13.733412 26304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7: Sending DeleteTablet for 3 replicas of tablet 8160815e9bb942278ae360db54e908b5
I20260928 14:48:13.733659 26431 tablet_service.cc:1514] Processing DeleteTablet for tablet 8160815e9bb942278ae360db54e908b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:37000
I20260928 14:48:13.733685 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet 8160815e9bb942278ae360db54e908b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:36326
I20260928 14:48:13.733768 26841 tablet_replica.cc:330] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: stopping tablet replica
I20260928 14:48:13.733824 26841 raft_consensus.cc:2229] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:13.733861 26841 raft_consensus.cc:2258] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.733929 26840 tablet_replica.cc:330] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: stopping tablet replica
I20260928 14:48:13.733994 26840 raft_consensus.cc:2229] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:13.734032 26840 raft_consensus.cc:2258] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.734331 26841 ts_tablet_manager.cc:1905] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.734382 26840 ts_tablet_manager.cc:1905] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.736034 26841 ts_tablet_manager.cc:1918] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:48:13.736099 26841 log.cc:1196] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/8160815e9bb942278ae360db54e908b5
I20260928 14:48:13.736263 26840 ts_tablet_manager.cc:1918] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:48:13.736317 26840 log.cc:1196] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/8160815e9bb942278ae360db54e908b5
I20260928 14:48:13.736320 26841 ts_tablet_manager.cc:1939] T 8160815e9bb942278ae360db54e908b5 P 2e341f6d73d8425fb4dca59f99d6d569: Deleting consensus metadata
I20260928 14:48:13.736577 26840 ts_tablet_manager.cc:1939] T 8160815e9bb942278ae360db54e908b5 P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting consensus metadata
I20260928 14:48:13.736974 26289 catalog_manager.cc:4841] TS 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641): tablet 8160815e9bb942278ae360db54e908b5 (table test-table [id=e6fb92d01f0142b6b344d95a79596f56]) successfully deleted
I20260928 14:48:13.737128 26290 catalog_manager.cc:4841] TS 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881): tablet 8160815e9bb942278ae360db54e908b5 (table test-table [id=e6fb92d01f0142b6b344d95a79596f56]) successfully deleted
I20260928 14:48:13.737290 26582 tablet_service.cc:1514] Processing DeleteTablet for tablet 8160815e9bb942278ae360db54e908b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:47548
I20260928 14:48:13.737422 26839 tablet_replica.cc:330] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: stopping tablet replica
I20260928 14:48:13.737488 26839 raft_consensus.cc:2229] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:48:13.737568 26839 raft_consensus.cc:2258] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.737824 26839 ts_tablet_manager.cc:1905] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.739413 26839 ts_tablet_manager.cc:1918] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:48:13.739494 26839 log.cc:1196] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/8160815e9bb942278ae360db54e908b5
I20260928 14:48:13.739720 26839 ts_tablet_manager.cc:1939] T 8160815e9bb942278ae360db54e908b5 P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting consensus metadata
I20260928 14:48:13.740051 26291 catalog_manager.cc:4841] TS 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130:39197): tablet 8160815e9bb942278ae360db54e908b5 (table test-table [id=e6fb92d01f0142b6b344d95a79596f56]) successfully deleted
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:48:13.764818 26355 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:13.758077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.18.96.190:42625)
0928 14:48:13.758139 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.758141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.758284 (+ 143us) server_negotiation.cc:407] Connection header received
0928 14:48:13.758324 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.758325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.758339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.758441 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:13.758995 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.759940 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.760710 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.761026 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.761106 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.764742 (+ 3636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42124: BlockingRecv error: recv got EOF from 127.0.0.1:42124 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 14:48:13 dist-test-slave-j968 krb5kdc[26260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM
I20260928 14:48:13.774010 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.765836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.18.96.190:42625)
0928 14:48:13.765881 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.765883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.766026 (+ 143us) server_negotiation.cc:407] Connection header received
0928 14:48:13.766064 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.766065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.766077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.766153 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:13.766738 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.767578 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.768405 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.768720 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.768806 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.771065 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:13.771067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:13.771067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:13.771076 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:13.772827 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.773195 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.773196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.773197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.773265 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:13.773469 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:13.773470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:13.773471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:13.773622 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:13.773735 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.773816 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.773900 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:48:13.775774 26304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:13.776070 26304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:13.777580 26431 tablet_service.cc:1467] Processing CreateTablet for tablet d1d2dc1b98ad4b4192c48daf8e54c093 (DEFAULT_TABLE table=test-table [id=b29df4bd78bd49b499969a4602a1ece4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.777607 26582 tablet_service.cc:1467] Processing CreateTablet for tablet d1d2dc1b98ad4b4192c48daf8e54c093 (DEFAULT_TABLE table=test-table [id=b29df4bd78bd49b499969a4602a1ece4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.777750 26431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d2dc1b98ad4b4192c48daf8e54c093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.777756 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d2dc1b98ad4b4192c48daf8e54c093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.779209 26817 tablet_bootstrap.cc:492] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: Bootstrap starting.
I20260928 14:48:13.779260 26722 tablet_service.cc:1467] Processing CreateTablet for tablet d1d2dc1b98ad4b4192c48daf8e54c093 (DEFAULT_TABLE table=test-table [id=b29df4bd78bd49b499969a4602a1ece4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.779390 26722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d2dc1b98ad4b4192c48daf8e54c093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.779650 26817 tablet_bootstrap.cc:654] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.780323 26817 tablet_bootstrap.cc:492] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: No bootstrap required, opened a new log
I20260928 14:48:13.780386 26817 ts_tablet_manager.cc:1397] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:13.780546 26817 raft_consensus.cc:348] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.780625 26817 raft_consensus.cc:374] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.780652 26817 raft_consensus.cc:729] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e341f6d73d8425fb4dca59f99d6d569, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.780691 26817 consensus_queue.cc:260] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.780789 26817 ts_tablet_manager.cc:1428] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.780843 26822 tablet_bootstrap.cc:492] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: Bootstrap starting.
I20260928 14:48:13.781181 26822 tablet_bootstrap.cc:654] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.781534 26816 tablet_bootstrap.cc:492] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: Bootstrap starting.
I20260928 14:48:13.781973 26816 tablet_bootstrap.cc:654] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.782969 26822 tablet_bootstrap.cc:492] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: No bootstrap required, opened a new log
I20260928 14:48:13.783030 26822 ts_tablet_manager.cc:1397] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:48:13.783146 26822 raft_consensus.cc:348] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.783231 26822 raft_consensus.cc:374] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.783252 26822 raft_consensus.cc:729] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e7132ccc2b4d08a7b2076426d0e7de, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.783300 26822 consensus_queue.cc:260] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.783416 26822 ts_tablet_manager.cc:1428] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:13.783715 26816 tablet_bootstrap.cc:492] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: No bootstrap required, opened a new log
I20260928 14:48:13.783769 26816 ts_tablet_manager.cc:1397] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:48:13.783962 26816 raft_consensus.cc:348] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.784049 26816 raft_consensus.cc:374] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.784072 26816 raft_consensus.cc:729] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d03ab2e91b4c5cb38a7b3da6d95505, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.784116 26816 consensus_queue.cc:260] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.784216 26816 ts_tablet_manager.cc:1428] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:13.789091 26821 raft_consensus.cc:482] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:13.789165 26821 raft_consensus.cc:504] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.789513 26821 leader_election.cc:288] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130:39197), 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641)
I20260928 14:48:13.792909 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.789889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:48579 (local address 127.18.96.130:39197)
0928 14:48:13.789936 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.789938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.789946 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:48:13.789972 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.789973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.789982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.790048 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:13.790224 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.790712 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.792171 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.792537 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.792633 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.792644 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.792805 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.792841 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:48:13.793192 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.789669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39197 (local address 127.18.96.129:48579)
0928 14:48:13.789787 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.789799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.789850 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.790055 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.790056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.790057 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:13.790135 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.790138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.790819 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.790820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.792091 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.792094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.792635 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.792636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.792717 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.792726 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.792796 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.792844 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260928 14:48:13.793470 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2dc1b98ad4b4192c48daf8e54c093" candidate_uuid: "2e341f6d73d8425fb4dca59f99d6d569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" is_pre_election: true
I20260928 14:48:13.793612 26602 raft_consensus.cc:2454] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e341f6d73d8425fb4dca59f99d6d569 in term 0.
I20260928 14:48:13.793795 26395 leader_election.cc:302] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e7132ccc2b4d08a7b2076426d0e7de, 2e341f6d73d8425fb4dca59f99d6d569; no voters:
I20260928 14:48:13.793920 26821 raft_consensus.cc:2781] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:13.793957 26821 raft_consensus.cc:482] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:13.793983 26821 raft_consensus.cc:3037] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.794322 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.789673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:41641 (local address 127.18.96.129:39751)
0928 14:48:13.790117 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.790128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.790186 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.790942 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.790943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.790946 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:13.791034 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.791036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.791949 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.791950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.793153 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.793156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.794083 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.794084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.794166 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.794170 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.794220 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.794255 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":58,"threads_started":1}
I20260928 14:48:13.794553 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.789775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:39751 (local address 127.18.96.131:41641)
0928 14:48:13.789913 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.789915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.790808 (+ 893us) server_negotiation.cc:407] Connection header received
0928 14:48:13.790835 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.790836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.790848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.790910 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:13.791116 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.791849 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.793384 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.793952 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.794023 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.794032 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.794441 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.794483 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260928 14:48:13.794828 26742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2dc1b98ad4b4192c48daf8e54c093" candidate_uuid: "2e341f6d73d8425fb4dca59f99d6d569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" is_pre_election: true
I20260928 14:48:13.794911 26742 raft_consensus.cc:2454] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e341f6d73d8425fb4dca59f99d6d569 in term 0.
I20260928 14:48:13.794927 26821 raft_consensus.cc:504] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.795050 26821 leader_election.cc:288] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [CANDIDATE]: Term 1 election: Requested vote from peers 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130:39197), 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641)
I20260928 14:48:13.795190 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2dc1b98ad4b4192c48daf8e54c093" candidate_uuid: "2e341f6d73d8425fb4dca59f99d6d569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20e7132ccc2b4d08a7b2076426d0e7de"
I20260928 14:48:13.795254 26602 raft_consensus.cc:3037] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.795307 26742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2dc1b98ad4b4192c48daf8e54c093" candidate_uuid: "2e341f6d73d8425fb4dca59f99d6d569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505"
I20260928 14:48:13.795385 26742 raft_consensus.cc:3037] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.796056 26602 raft_consensus.cc:2454] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e341f6d73d8425fb4dca59f99d6d569 in term 1.
I20260928 14:48:13.796213 26742 raft_consensus.cc:2454] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e341f6d73d8425fb4dca59f99d6d569 in term 1.
I20260928 14:48:13.796221 26395 leader_election.cc:302] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e7132ccc2b4d08a7b2076426d0e7de, 2e341f6d73d8425fb4dca59f99d6d569; no voters:
I20260928 14:48:13.796322 26821 raft_consensus.cc:2781] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:13.796481 26821 raft_consensus.cc:686] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 LEADER]: Becoming Leader. State: Replica: 2e341f6d73d8425fb4dca59f99d6d569, State: Running, Role: LEADER
I20260928 14:48:13.796533 26821 consensus_queue.cc:237] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.797260 26304 catalog_manager.cc:5495] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129). New cstate: current_term: 1 leader_uuid: "2e341f6d73d8425fb4dca59f99d6d569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } health_report { overall_health: UNKNOWN } } }
I20260928 14:48:13.803614 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.800903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.18.96.129:32881)
0928 14:48:13.801062 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.801064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.801070 (+ 6us) server_negotiation.cc:407] Connection header received
0928 14:48:13.801102 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.801103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.801119 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.801202 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:13.801355 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.801883 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.802748 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.802969 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.803043 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.803222 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.803332 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.803446 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.803487 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.803508 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260928 14:48:13.804554 26742 raft_consensus.cc:1261] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Refusing update from remote peer 2e341f6d73d8425fb4dca59f99d6d569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.804708 26602 raft_consensus.cc:1261] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Refusing update from remote peer 2e341f6d73d8425fb4dca59f99d6d569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.804831 26886 consensus_queue.cc:1035] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.804943 26821 consensus_queue.cc:1035] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.808564 26305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:13.808668 26305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:13.809300 26305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7: Sending DeleteTablet for 3 replicas of tablet d1d2dc1b98ad4b4192c48daf8e54c093
I20260928 14:48:13.809579 26431 tablet_service.cc:1514] Processing DeleteTablet for tablet d1d2dc1b98ad4b4192c48daf8e54c093 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:37000
I20260928 14:48:13.809677 26841 tablet_replica.cc:330] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: stopping tablet replica
I20260928 14:48:13.809723 26841 raft_consensus.cc:2229] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:48:13.809804 26841 raft_consensus.cc:2258] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.809885 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet d1d2dc1b98ad4b4192c48daf8e54c093 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:36326
I20260928 14:48:13.809937 26582 tablet_service.cc:1514] Processing DeleteTablet for tablet d1d2dc1b98ad4b4192c48daf8e54c093 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:47548
I20260928 14:48:13.810021 26839 tablet_replica.cc:330] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: stopping tablet replica
I20260928 14:48:13.810030 26840 tablet_replica.cc:330] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: stopping tablet replica
I20260928 14:48:13.810062 26839 raft_consensus.cc:2229] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:13.810078 26840 raft_consensus.cc:2229] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:13.810104 26839 raft_consensus.cc:2258] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.810122 26840 raft_consensus.cc:2258] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.810268 26305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:13.810408 26305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:13.811102 26841 ts_tablet_manager.cc:1905] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.812351 26839 ts_tablet_manager.cc:1905] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.812839 26841 ts_tablet_manager.cc:1918] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:13.812896 26841 log.cc:1196] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/d1d2dc1b98ad4b4192c48daf8e54c093
I20260928 14:48:13.813133 26841 ts_tablet_manager.cc:1939] T d1d2dc1b98ad4b4192c48daf8e54c093 P 2e341f6d73d8425fb4dca59f99d6d569: Deleting consensus metadata
I20260928 14:48:13.813428 26840 ts_tablet_manager.cc:1905] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.813865 26839 ts_tablet_manager.cc:1918] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:13.813916 26839 log.cc:1196] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/d1d2dc1b98ad4b4192c48daf8e54c093
I20260928 14:48:13.814133 26839 ts_tablet_manager.cc:1939] T d1d2dc1b98ad4b4192c48daf8e54c093 P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting consensus metadata
I20260928 14:48:13.814894 26291 catalog_manager.cc:4841] TS 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130:39197): tablet d1d2dc1b98ad4b4192c48daf8e54c093 (table test-table [id=b29df4bd78bd49b499969a4602a1ece4]) successfully deleted
I20260928 14:48:13.814983 26840 ts_tablet_manager.cc:1918] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:13.815049 26840 log.cc:1196] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/d1d2dc1b98ad4b4192c48daf8e54c093
I20260928 14:48:13.815284 26840 ts_tablet_manager.cc:1939] T d1d2dc1b98ad4b4192c48daf8e54c093 P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting consensus metadata
I20260928 14:48:13.815678 26289 catalog_manager.cc:4841] TS 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641): tablet d1d2dc1b98ad4b4192c48daf8e54c093 (table test-table [id=b29df4bd78bd49b499969a4602a1ece4]) successfully deleted
I20260928 14:48:13.816035 26431 tablet_service.cc:1467] Processing CreateTablet for tablet e066e523cd1c4668a80f723e7324251c (DEFAULT_TABLE table=test-table [id=58a77b20f0f64ef9a7934f8572518512]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.816157 26431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e066e523cd1c4668a80f723e7324251c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.817410 26817 tablet_bootstrap.cc:492] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569: Bootstrap starting.
I20260928 14:48:13.817834 26817 tablet_bootstrap.cc:654] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.818655 26722 tablet_service.cc:1467] Processing CreateTablet for tablet e066e523cd1c4668a80f723e7324251c (DEFAULT_TABLE table=test-table [id=58a77b20f0f64ef9a7934f8572518512]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.818768 26722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e066e523cd1c4668a80f723e7324251c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.819972 26816 tablet_bootstrap.cc:492] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: Bootstrap starting.
I20260928 14:48:13.820621 26816 tablet_bootstrap.cc:654] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.821236 26816 tablet_bootstrap.cc:492] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: No bootstrap required, opened a new log
I20260928 14:48:13.821282 26816 ts_tablet_manager.cc:1397] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:13.821273 26582 tablet_service.cc:1467] Processing CreateTablet for tablet e066e523cd1c4668a80f723e7324251c (DEFAULT_TABLE table=test-table [id=58a77b20f0f64ef9a7934f8572518512]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:13.821378 26582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e066e523cd1c4668a80f723e7324251c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:13.821452 26816 raft_consensus.cc:348] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.821518 26816 raft_consensus.cc:374] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.821539 26816 raft_consensus.cc:729] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d03ab2e91b4c5cb38a7b3da6d95505, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.821592 26816 consensus_queue.cc:260] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.821676 26816 ts_tablet_manager.cc:1428] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.822605 26822 tablet_bootstrap.cc:492] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: Bootstrap starting.
I20260928 14:48:13.822897 26817 tablet_bootstrap.cc:492] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569: No bootstrap required, opened a new log
I20260928 14:48:13.822942 26817 ts_tablet_manager.cc:1397] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 14:48:13.823060 26822 tablet_bootstrap.cc:654] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:13.823092 26817 raft_consensus.cc:348] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.823176 26817 raft_consensus.cc:374] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.823202 26817 raft_consensus.cc:729] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e341f6d73d8425fb4dca59f99d6d569, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.823239 26817 consensus_queue.cc:260] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.823318 26817 ts_tablet_manager.cc:1428] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.823823 26822 tablet_bootstrap.cc:492] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: No bootstrap required, opened a new log
I20260928 14:48:13.823958 26822 ts_tablet_manager.cc:1397] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:13.824106 26822 raft_consensus.cc:348] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.824170 26822 raft_consensus.cc:374] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:13.824191 26822 raft_consensus.cc:729] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e7132ccc2b4d08a7b2076426d0e7de, State: Initialized, Role: FOLLOWER
I20260928 14:48:13.824232 26822 consensus_queue.cc:260] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.824313 26822 ts_tablet_manager.cc:1428] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:13.824493 26290 catalog_manager.cc:4841] TS 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881): tablet d1d2dc1b98ad4b4192c48daf8e54c093 (table test-table [id=b29df4bd78bd49b499969a4602a1ece4]) successfully deleted
I20260928 14:48:13.904479 26820 raft_consensus.cc:482] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:13.904608 26820 raft_consensus.cc:504] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.905011 26820 leader_election.cc:288] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881), 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130:39197)
I20260928 14:48:13.908761 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.905334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:48083 (local address 127.18.96.130:39197)
0928 14:48:13.905399 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.905401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.905549 (+ 148us) server_negotiation.cc:407] Connection header received
0928 14:48:13.905630 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.905631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.905640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.905686 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:13.905897 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.906743 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.908001 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.908357 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.908447 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.908458 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.908635 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.908682 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 14:48:13.908828 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.905211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39197 (local address 127.18.96.131:48083)
0928 14:48:13.905471 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.905484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.905568 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.905726 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.905727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.905729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:13.905828 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.905831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.906868 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.906869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.907915 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.907917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.908475 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.908476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.908544 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.908549 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.908607 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.908670 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":26,"thread_start_us":62,"threads_started":1}
I20260928 14:48:13.909008 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.905382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:54601 (local address 127.18.96.129:32881)
0928 14:48:13.905445 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.905447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.905521 (+ 74us) server_negotiation.cc:407] Connection header received
0928 14:48:13.905611 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.905613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.905623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.905672 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:13.905838 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.906335 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.907765 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.908387 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.908470 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.908481 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.908884 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.908929 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:48:13.909278 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.905211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:32881 (local address 127.18.96.131:54601)
0928 14:48:13.905460 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.905472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.905538 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.905685 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.905686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.905689 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:13.905751 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.905754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.906433 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.906434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.907642 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.907644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.908504 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.908505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.908555 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.908558 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.908608 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.908670 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260928 14:48:13.909228 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e066e523cd1c4668a80f723e7324251c" candidate_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" is_pre_election: true
I20260928 14:48:13.909348 26602 raft_consensus.cc:2454] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d03ab2e91b4c5cb38a7b3da6d95505 in term 0.
I20260928 14:48:13.909551 26675 leader_election.cc:302] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20e7132ccc2b4d08a7b2076426d0e7de, 67d03ab2e91b4c5cb38a7b3da6d95505; no voters:
I20260928 14:48:13.909574 26462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e066e523cd1c4668a80f723e7324251c" candidate_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e341f6d73d8425fb4dca59f99d6d569" is_pre_election: true
I20260928 14:48:13.909663 26462 raft_consensus.cc:2454] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d03ab2e91b4c5cb38a7b3da6d95505 in term 0.
I20260928 14:48:13.909693 26820 raft_consensus.cc:2781] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:13.909725 26820 raft_consensus.cc:482] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:13.909747 26820 raft_consensus.cc:3037] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.910758 26820 raft_consensus.cc:504] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.910910 26820 leader_election.cc:288] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [CANDIDATE]: Term 1 election: Requested vote from peers 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881), 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130:39197)
I20260928 14:48:13.911211 26462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e066e523cd1c4668a80f723e7324251c" candidate_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e341f6d73d8425fb4dca59f99d6d569"
I20260928 14:48:13.911227 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e066e523cd1c4668a80f723e7324251c" candidate_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20e7132ccc2b4d08a7b2076426d0e7de"
I20260928 14:48:13.911301 26462 raft_consensus.cc:3037] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.911312 26602 raft_consensus.cc:3037] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:13.911976 26462 raft_consensus.cc:2454] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d03ab2e91b4c5cb38a7b3da6d95505 in term 1.
I20260928 14:48:13.912150 26674 leader_election.cc:302] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e341f6d73d8425fb4dca59f99d6d569, 67d03ab2e91b4c5cb38a7b3da6d95505; no voters:
I20260928 14:48:13.912231 26602 raft_consensus.cc:2454] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d03ab2e91b4c5cb38a7b3da6d95505 in term 1.
I20260928 14:48:13.912424 26820 raft_consensus.cc:2781] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:13.912595 26820 raft_consensus.cc:686] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 LEADER]: Becoming Leader. State: Replica: 67d03ab2e91b4c5cb38a7b3da6d95505, State: Running, Role: LEADER
I20260928 14:48:13.912689 26820 consensus_queue.cc:237] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } }
I20260928 14:48:13.913496 26305 catalog_manager.cc:5495] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131). New cstate: current_term: 1 leader_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d03ab2e91b4c5cb38a7b3da6d95505" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 41641 } health_report { overall_health: HEALTHY } } }
I20260928 14:48:13.940662 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.937974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.18.96.131:41641)
0928 14:48:13.938040 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.938043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.938052 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:13.938095 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.938096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.938136 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.938204 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:13.938307 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.938812 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.939696 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.939990 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.940079 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.940271 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.940399 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.940495 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.940541 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.940566 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:48:13.945801 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.943524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:55689 (local address 127.18.96.130:39197)
0928 14:48:13.943571 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.943573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.943673 (+ 100us) server_negotiation.cc:407] Connection header received
0928 14:48:13.943713 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.943713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.943722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.943760 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:13.943908 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.944385 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.945185 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.945367 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.945416 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.945575 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.945601 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.945640 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.945728 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.945745 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:48:13.945849 26902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.943387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39197 (local address 127.18.96.131:55689)
0928 14:48:13.943596 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.943607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.943666 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.943777 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.943778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.943780 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:48:13.943832 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.943833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.944467 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.944468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.945090 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.945092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.945435 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.945436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.945488 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.945501 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.945675 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.945677 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.945728 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.945768 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
I20260928 14:48:13.948042 26462 raft_consensus.cc:1261] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Refusing update from remote peer 67d03ab2e91b4c5cb38a7b3da6d95505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.948041 26602 raft_consensus.cc:1261] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Refusing update from remote peer 67d03ab2e91b4c5cb38a7b3da6d95505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:13.948294 26898 consensus_queue.cc:1035] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20e7132ccc2b4d08a7b2076426d0e7de" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.948408 26820 consensus_queue.cc:1035] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e341f6d73d8425fb4dca59f99d6d569" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:13.955068 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.952804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:41641 (local address 127.18.96.130:55423)
0928 14:48:13.952864 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.952877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:13.952927 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:13.953089 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:13.953091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:13.953092 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:48:13.953141 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.953143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.953854 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.953855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.954398 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:13.954399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.954716 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.954716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:13.954755 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.954760 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.954900 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.954901 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 14:48:13.954945 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:48:13.954992 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:48:13.955088 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.952842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:55423 (local address 127.18.96.131:41641)
0928 14:48:13.952883 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:13.952885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:13.952912 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:48:13.952984 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:13.952985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:13.952996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:13.953046 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:13.953224 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.953790 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.954467 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:13.954656 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:13.954704 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:13.954820 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:13.954853 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:13.954898 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:13.954980 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 14:48:13.954997 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:48:13.962212 26305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:13.962332 26305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:13.963040 26305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0c7cd12e194907976a69ecedfe9de7: Sending DeleteTablet for 3 replicas of tablet e066e523cd1c4668a80f723e7324251c
I20260928 14:48:13.963306 26431 tablet_service.cc:1514] Processing DeleteTablet for tablet e066e523cd1c4668a80f723e7324251c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:37000
I20260928 14:48:13.963349 26582 tablet_service.cc:1514] Processing DeleteTablet for tablet e066e523cd1c4668a80f723e7324251c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:47548
I20260928 14:48:13.963378 26722 tablet_service.cc:1514] Processing DeleteTablet for tablet e066e523cd1c4668a80f723e7324251c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:13 UTC) from {username='oryx', principal='oryx/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:36326
I20260928 14:48:13.963469 26841 tablet_replica.cc:330] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569: stopping tablet replica
I20260928 14:48:13.963471 26840 tablet_replica.cc:330] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: stopping tablet replica
I20260928 14:48:13.963522 26841 raft_consensus.cc:2229] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:13.963526 26840 raft_consensus.cc:2229] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:48:13.963568 26841 raft_consensus.cc:2258] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.963603 26840 raft_consensus.cc:2258] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.963647 26839 tablet_replica.cc:330] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: stopping tablet replica
I20260928 14:48:13.963719 26839 raft_consensus.cc:2229] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:13.963761 26839 raft_consensus.cc:2258] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:13.963954 26840 ts_tablet_manager.cc:1905] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.964119 26841 ts_tablet_manager.cc:1905] T e066e523cd1c4668a80f723e7324251c P 2e341f6d73d8425fb4dca59f99d6d569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.964433 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 26377
I20260928 14:48:13.966033 26840 ts_tablet_manager.cc:1918] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:48:13.966094 26840 log.cc:1196] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/e066e523cd1c4668a80f723e7324251c
I20260928 14:48:13.966317 26840 ts_tablet_manager.cc:1939] T e066e523cd1c4668a80f723e7324251c P 67d03ab2e91b4c5cb38a7b3da6d95505: Deleting consensus metadata
I20260928 14:48:13.966714 26289 catalog_manager.cc:4841] TS 67d03ab2e91b4c5cb38a7b3da6d95505 (127.18.96.131:41641): tablet e066e523cd1c4668a80f723e7324251c (table test-table [id=58a77b20f0f64ef9a7934f8572518512]) successfully deleted
I20260928 14:48:13.968261 26839 ts_tablet_manager.cc:1905] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:13.969892 26839 ts_tablet_manager.cc:1918] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:48:13.969954 26839 log.cc:1196] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/e066e523cd1c4668a80f723e7324251c
I20260928 14:48:13.970156 26839 ts_tablet_manager.cc:1939] T e066e523cd1c4668a80f723e7324251c P 20e7132ccc2b4d08a7b2076426d0e7de: Deleting consensus metadata
I20260928 14:48:13.970518 26291 catalog_manager.cc:4841] TS 20e7132ccc2b4d08a7b2076426d0e7de (127.18.96.130:39197): tablet e066e523cd1c4668a80f723e7324251c (table test-table [id=58a77b20f0f64ef9a7934f8572518512]) successfully deleted
W20260928 14:48:13.974824 26290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:32881 (error 108)
I20260928 14:48:13.975090 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 26517
I20260928 14:48:13.980173 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:13.975228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:32881 (local address 127.0.0.1:37018)
0928 14:48:13.980056 (+ 4828us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:13.980133 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4801,"thread_start_us":43,"threads_started":1}
W20260928 14:48:13.980309 26290 catalog_manager.cc:4584] TS 2e341f6d73d8425fb4dca59f99d6d569 (127.18.96.129:32881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e066e523cd1c4668a80f723e7324251c: Network error: Client connection negotiation failed: client connection to 127.18.96.129:32881: connect: Connection refused (error 111)
I20260928 14:48:13.982913 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 26657
I20260928 14:48:13.988739 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 26273
2026-09-28T14:48:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2880 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:14 dist-test-slave-j968 krb5kdc[26911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:14 dist-test-slave-j968 krb5kdc[26911](info): set up 2 sockets
Sep 28 14:48:14 dist-test-slave-j968 krb5kdc[26911](info): commencing operation
krb5kdc: starting...
W20260928 14:48:15.045589 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:48:15 dist-test-slave-j968 krb5kdc[26911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:48:15.169194 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:40287
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:40287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:15.252147 26924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:15.252365 26924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:15.252388 26924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:15.253602 26924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:15.253659 26924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:15.253684 26924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:15.253705 26924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:15.253737 26924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:15.255261 26924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:40287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:40287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:15.255529 26924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:15.255774 26924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:15.260751 26932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:15.260833 26924 server_base.cc:1034] running on GCE node
W20260928 14:48:15.260759 26929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:15.260772 26930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:15.261325 26924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:15.261619 26924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:15.262789 26924 hybrid_clock.cc:648] HybridClock initialized: now 1790606895262775 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T14:48:15Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1276 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:15 dist-test-slave-j968 krb5kdc[26937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:15 dist-test-slave-j968 krb5kdc[26937](info): set up 2 sockets
Sep 28 14:48:15 dist-test-slave-j968 krb5kdc[26937](info): commencing operation
krb5kdc: starting...
W20260928 14:48:16.305631 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:48:16 dist-test-slave-j968 krb5kdc[26937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:48:16.427310 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34945
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:34945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:16.509927 26950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:16.510169 26950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:16.510205 26950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:16.511328 26950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:16.511401 26950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:16.511426 26950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:16.511462 26950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:16.511493 26950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:16.513020 26950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:34945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:16.513303 26950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:16.513525 26950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:16.518791 26955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:16.518833 26956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:16.518795 26958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:16.519287 26950 server_base.cc:1034] running on GCE node
I20260928 14:48:16.519503 26950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:16.519752 26950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:16.520902 26950 hybrid_clock.cc:648] HybridClock initialized: now 1790606896520874 us; error 40 us; skew 500 ppm
Sep 28 14:48:16 dist-test-slave-j968 krb5kdc[26937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606896, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:16.523357 26950 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:48:16.523830 26950 webserver.cc:460] Webserver started at http://127.18.96.190:36951/ using document root <none> and password file <none>
I20260928 14:48:16.524097 26950 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:16.524142 26950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:16.524231 26950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:16.525279 26950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "7ec0fc3442984433ab2a0cad81b3deac"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "34f07c1e382ff37385ee7afc43d07248"
server_key_iv: "1e60a52f41d9a6a7ad763cc26333d66b"
server_key_version: "encryptionkey@0"
I20260928 14:48:16.525585 26950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "7ec0fc3442984433ab2a0cad81b3deac"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "34f07c1e382ff37385ee7afc43d07248"
server_key_iv: "1e60a52f41d9a6a7ad763cc26333d66b"
server_key_version: "encryptionkey@0"
I20260928 14:48:16.527048 26950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:16.527819 26965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:16.528118 26950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:16.528239 26950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "7ec0fc3442984433ab2a0cad81b3deac"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "34f07c1e382ff37385ee7afc43d07248"
server_key_iv: "1e60a52f41d9a6a7ad763cc26333d66b"
server_key_version: "encryptionkey@0"
I20260928 14:48:16.528317 26950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:16.543363 26950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:16.545374 26950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:16.545555 26950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:16.549763 26950 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:34945
I20260928 14:48:16.549801 27017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:34945 every 8 connection(s)
I20260928 14:48:16.550184 26950 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:16.551057 27018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:16.552161 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 26950
I20260928 14:48:16.552286 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:16.552498 18818 external_mini_cluster.cc:1428] Setting key 1eda56341205d959afc450d669fa5862
I20260928 14:48:16.553957 27018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: Bootstrap starting.
I20260928 14:48:16.554664 27018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:16.555019 27018 log.cc:824] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:16.555828 27018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: No bootstrap required, opened a new log
I20260928 14:48:16.558116 27018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } }
I20260928 14:48:16.558307 27018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:16.558346 27018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec0fc3442984433ab2a0cad81b3deac, State: Initialized, Role: FOLLOWER
I20260928 14:48:16.558516 27018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } }
I20260928 14:48:16.558637 27018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:16.558683 27018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:16.558738 27018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:16.559283 27018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } }
I20260928 14:48:16.559417 27018 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec0fc3442984433ab2a0cad81b3deac; no voters:
I20260928 14:48:16.559602 27018 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:16.559646 27023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:16.559793 27023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 1 LEADER]: Becoming Leader. State: Replica: 7ec0fc3442984433ab2a0cad81b3deac, State: Running, Role: LEADER
I20260928 14:48:16.559865 27023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } }
I20260928 14:48:16.559901 27018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:16.560273 27024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ec0fc3442984433ab2a0cad81b3deac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } } }
I20260928 14:48:16.560271 27025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec0fc3442984433ab2a0cad81b3deac. Latest consensus state: current_term: 1 leader_uuid: "7ec0fc3442984433ab2a0cad81b3deac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } } }
I20260928 14:48:16.560374 27024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: This master's current role is: LEADER
Sep 28 14:48:16 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:16.560379 27025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:16.560804 27032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:16.561548 27032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:16.563287 27032 catalog_manager.cc:1347] Generated new cluster ID: fcd59823ca084c07b3cb7578cca85c16
I20260928 14:48:16.563339 27032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:16.565268 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:16.553241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.18.96.190:34945)
0928 14:48:16.553445 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:16.553455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:16.553491 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:48:16.554047 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:16.554057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:16.554130 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:16.554300 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:16.554934 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.556348 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.557138 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.557837 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.557961 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:16.561014 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:16.561019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:16.561019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:16.561031 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:16.563598 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:16.564048 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:16.564052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:16.564052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:16.564162 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:16.564395 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:16.564396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:16.564396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:16.564648 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:16.564814 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:16.564926 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 14:48:16.565068 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260928 14:48:16.570442 27032 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:16.571156 27032 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:16.575055 27032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: Generated new TSK 0
I20260928 14:48:16.575235 27032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:48:16.604843 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34945
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:16.701236 27044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:16.701447 27044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:16.701470 27044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:16.702646 27044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:16.702685 27044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:16.702711 27044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:16.704202 27044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:16.704444 27044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:16.704684 27044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:16.709877 27052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:16.709887 27049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:16.709892 27050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:16.710194 27044 server_base.cc:1034] running on GCE node
I20260928 14:48:16.710404 27044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:16.710654 27044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:16.711813 27044 hybrid_clock.cc:648] HybridClock initialized: now 1790606896711800 us; error 30 us; skew 500 ppm
Sep 28 14:48:16 dist-test-slave-j968 krb5kdc[26937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606896, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:16.714630 27044 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:48:16.715132 27044 webserver.cc:460] Webserver started at http://127.18.96.129:37281/ using document root <none> and password file <none>
I20260928 14:48:16.715401 27044 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:16.715466 27044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:16.715598 27044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:16.716979 27044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "8ce62bf5ce9e43cebb485643ce247013"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "1274a1d12d6695e97f358d8c37eab9c1"
server_key_iv: "7e01c4cf54fc3ab90782ec7ac0dc08d3"
server_key_version: "encryptionkey@0"
I20260928 14:48:16.717350 27044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "8ce62bf5ce9e43cebb485643ce247013"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "1274a1d12d6695e97f358d8c37eab9c1"
server_key_iv: "7e01c4cf54fc3ab90782ec7ac0dc08d3"
server_key_version: "encryptionkey@0"
I20260928 14:48:16.718927 27044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:16.719791 27059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:16.719985 27044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:16.720072 27044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "8ce62bf5ce9e43cebb485643ce247013"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "1274a1d12d6695e97f358d8c37eab9c1"
server_key_iv: "7e01c4cf54fc3ab90782ec7ac0dc08d3"
server_key_version: "encryptionkey@0"
I20260928 14:48:16.720165 27044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:16.741341 27044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:16.743459 27044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:16.743578 27044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:16.743774 27044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:16.744158 27044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:16.744195 27044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:16.744220 27044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:16.744231 27044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:16.750427 27044 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:33575
I20260928 14:48:16.750522 27172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:33575 every 8 connection(s)
I20260928 14:48:16.750844 27044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:48:16 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606896, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:16.760743 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:16.751665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:34217 (local address 127.18.96.190:34945)
0928 14:48:16.751716 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:16.751718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:16.752017 (+ 299us) server_negotiation.cc:407] Connection header received
0928 14:48:16.752552 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:16.752554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:16.752565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:16.752618 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:16.753593 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.754115 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.755131 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.755347 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.755424 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:16.757720 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:16.757722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:16.757722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:16.757729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:16.759195 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:16.759612 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:16.759613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:16.759613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:16.759676 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:16.760000 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:16.760000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:16.760001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:16.760143 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:16.760201 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:16.760483 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 14:48:16.760612 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:48:16.760914 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 27044
I20260928 14:48:16.760953 27175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:16.751652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:34217)
0928 14:48:16.751842 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:16.751927 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:16.752441 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:16.752697 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:16.752701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:16.753279 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:16.753487 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:16.753492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.754219 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.754220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:16.755025 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:16.755026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.755432 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.755433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:16.755561 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:16.756146 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:16.756155 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:16.757616 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:16.759298 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:16.759304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:16.759306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:16.759536 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:16.759786 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:16.759787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:16.759787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:16.759896 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:16.760206 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:16.760212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:16.760331 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:48:16.760413 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:48:16.760656 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260928 14:48:16.761053 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:16.761291 18818 external_mini_cluster.cc:1428] Setting key 385e8bfb074cbfc3551fa7a61dc093eb
I20260928 14:48:16.761484 27173 heartbeater.cc:347] Connected to a master server at 127.18.96.190:34945
I20260928 14:48:16.761602 27173 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:16.761833 27173 heartbeater.cc:511] Master 127.18.96.190:34945 requested a full tablet report, sending...
I20260928 14:48:16.762485 26982 ts_manager.cc:194] Registered new tserver with Master: 8ce62bf5ce9e43cebb485643ce247013 (127.18.96.129:33575)
I20260928 14:48:16.763563 26982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:34217
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:48:16.787336 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34945
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:16.869125 27178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:16.869329 27178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:16.869352 27178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:16.870479 27178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:16.870518 27178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:16.870545 27178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:16.871970 27178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:16.872155 27178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:16.872319 27178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:16.877274 27183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:16.877280 27184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:16.877290 27186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:16.877650 27178 server_base.cc:1034] running on GCE node
I20260928 14:48:16.877873 27178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:16.878127 27178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:16.879276 27178 hybrid_clock.cc:648] HybridClock initialized: now 1790606896879266 us; error 31 us; skew 500 ppm
Sep 28 14:48:16 dist-test-slave-j968 krb5kdc[26937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606896, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:16.881816 27178 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:48:16.882273 27178 webserver.cc:460] Webserver started at http://127.18.96.130:46255/ using document root <none> and password file <none>
I20260928 14:48:16.882483 27178 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:16.882526 27178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:16.882617 27178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:16.883654 27178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "00253214a854481a9e525b530bf2fe05"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "1382970318fc582fe612d76d611f1cb2"
server_key_iv: "da93e1d11ea1e705e449f8adace19dc6"
server_key_version: "encryptionkey@0"
I20260928 14:48:16.884038 27178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "00253214a854481a9e525b530bf2fe05"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "1382970318fc582fe612d76d611f1cb2"
server_key_iv: "da93e1d11ea1e705e449f8adace19dc6"
server_key_version: "encryptionkey@0"
I20260928 14:48:16.885313 27178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:16.886065 27193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:16.886261 27178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:16.886369 27178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "00253214a854481a9e525b530bf2fe05"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "1382970318fc582fe612d76d611f1cb2"
server_key_iv: "da93e1d11ea1e705e449f8adace19dc6"
server_key_version: "encryptionkey@0"
I20260928 14:48:16.886443 27178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:16.905319 27178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:16.907226 27178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:16.907388 27178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:16.907617 27178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:16.907995 27178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:16.908051 27178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:16.908092 27178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:16.908128 27178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:16.913951 27178 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:37087
I20260928 14:48:16.914058 27306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:37087 every 8 connection(s)
I20260928 14:48:16.914372 27178 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:48:16 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606896, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:16.922691 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 27178
I20260928 14:48:16.922840 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:16.923060 18818 external_mini_cluster.cc:1428] Setting key 39a8bd2932d67205cc38fd474b353698
I20260928 14:48:16.924031 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:16.915133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:60131 (local address 127.18.96.190:34945)
0928 14:48:16.915177 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:16.915178 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:16.915416 (+ 238us) server_negotiation.cc:407] Connection header received
0928 14:48:16.916006 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:16.916007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:16.916017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:16.916070 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:16.916981 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.917469 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.918427 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.918651 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.918711 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:16.920962 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:16.920964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:16.920965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:16.920972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:16.922192 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:16.922778 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:16.922780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:16.922780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:16.922856 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:16.923150 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:16.923151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:16.923151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:16.923319 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:16.923389 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:16.923765 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 14:48:16.923867 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:48:16.924223 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:16.915117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:60131)
0928 14:48:16.915262 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:16.915299 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:16.915841 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:16.916126 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:16.916130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:16.916706 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:16.916897 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:16.916901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.917549 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.917550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:16.918335 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:16.918337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:16.918729 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:16.918730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:16.918835 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:16.919382 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:16.919389 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:16.920859 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:16.922298 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:16.922306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:16.922308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:16.922664 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:16.922944 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:16.922946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:16.922946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:16.923067 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:16.923399 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:16.923407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:16.923568 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 14:48:16.923715 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 14:48:16.923948 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 14:48:16.925622 27307 heartbeater.cc:347] Connected to a master server at 127.18.96.190:34945
I20260928 14:48:16.925741 27307 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:16.925931 27307 heartbeater.cc:511] Master 127.18.96.190:34945 requested a full tablet report, sending...
I20260928 14:48:16.926329 26982 ts_manager.cc:194] Registered new tserver with Master: 00253214a854481a9e525b530bf2fe05 (127.18.96.130:37087)
I20260928 14:48:16.926745 26982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:60131
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:48:16.950865 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34945
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:17.032848 27312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:17.033092 27312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:17.033120 27312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:17.034327 27312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:17.034375 27312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:17.034401 27312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:17.036021 27312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:17.036283 27312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:17.036507 27312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:17.041527 27317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:17.041579 27318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:17.041615 27312 server_base.cc:1034] running on GCE node
W20260928 14:48:17.041566 27320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:17.041898 27312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:17.042155 27312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:17.043299 27312 hybrid_clock.cc:648] HybridClock initialized: now 1790606897043285 us; error 28 us; skew 500 ppm
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606897, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:17.046020 27312 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:48:17.046469 27312 webserver.cc:460] Webserver started at http://127.18.96.131:36041/ using document root <none> and password file <none>
I20260928 14:48:17.046753 27312 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:17.046801 27312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:17.046893 27312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:17.048025 27312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "fcc480cab46b4937910fef077db87735"
format_stamp: "Formatted at 2026-09-28 14:48:17 on dist-test-slave-j968"
server_key: "0ca79ce1214bf34a7b49275d12db4e68"
server_key_iv: "fa8259b9f1372907a545caef7eb84bf7"
server_key_version: "encryptionkey@0"
I20260928 14:48:17.048336 27312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "fcc480cab46b4937910fef077db87735"
format_stamp: "Formatted at 2026-09-28 14:48:17 on dist-test-slave-j968"
server_key: "0ca79ce1214bf34a7b49275d12db4e68"
server_key_iv: "fa8259b9f1372907a545caef7eb84bf7"
server_key_version: "encryptionkey@0"
I20260928 14:48:17.049584 27312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:17.050259 27327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.050449 27312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:17.050539 27312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "fcc480cab46b4937910fef077db87735"
format_stamp: "Formatted at 2026-09-28 14:48:17 on dist-test-slave-j968"
server_key: "0ca79ce1214bf34a7b49275d12db4e68"
server_key_iv: "fa8259b9f1372907a545caef7eb84bf7"
server_key_version: "encryptionkey@0"
I20260928 14:48:17.050593 27312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:17.073554 27312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:17.075903 27312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:17.076035 27312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:17.076220 27312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:17.076548 27312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:17.076587 27312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.076645 27312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:17.076671 27312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.082501 27312 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:37005
I20260928 14:48:17.082666 27440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:37005 every 8 connection(s)
I20260928 14:48:17.082950 27312 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:17.085727 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 27312
I20260928 14:48:17.085816 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:17.086022 18818 external_mini_cluster.cc:1428] Setting key 268db6cb0b61d96051630d7738f16442
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606897, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:17.092532 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.083665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:43771 (local address 127.18.96.190:34945)
0928 14:48:17.083713 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.083716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.083934 (+ 218us) server_negotiation.cc:407] Connection header received
0928 14:48:17.084468 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.084469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.084482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.084529 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.085506 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.086191 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.087462 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.087651 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.087708 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.089854 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:17.089855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:17.089855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:17.089861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:17.091298 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:17.091662 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:17.091663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:17.091664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:17.091711 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:17.091952 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:17.091953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:17.091953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:17.092089 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:17.092144 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:17.092388 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 14:48:17.092468 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:48:17.092662 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.083629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:43771)
0928 14:48:17.083792 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.083828 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.084363 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.084598 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.084602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.085175 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.085376 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.085381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.086305 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.086307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.087382 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.087386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.087728 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.087729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.087838 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.088373 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.088381 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.089770 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:17.091369 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:17.091374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:17.091376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:17.091591 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:17.091765 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:17.091765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:17.091765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:17.091859 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:17.092152 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:17.092156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:17.092269 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 14:48:17.092347 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:48:17.092467 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260928 14:48:17.093186 27441 heartbeater.cc:347] Connected to a master server at 127.18.96.190:34945
I20260928 14:48:17.093281 27441 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:17.093418 27441 heartbeater.cc:511] Master 127.18.96.190:34945 requested a full tablet report, sending...
I20260928 14:48:17.093765 26982 ts_manager.cc:194] Registered new tserver with Master: fcc480cab46b4937910fef077db87735 (127.18.96.131:37005)
I20260928 14:48:17.094137 26982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:43771
I20260928 14:48:17.097743 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab.
Entry for principal oryx/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab.
WARNING: no policy specified for oryx/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab.
Entry for principal oryx/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab.
WARNING: no policy specified for oryx/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab.
Entry for principal oryx/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab.
I20260928 14:48:17.171655 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 27044
I20260928 14:48:17.176180 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 27178
I20260928 14:48:17.180579 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 27312
I20260928 14:48:17.184767 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 26950
I20260928 14:48:17.189944 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34945
--webserver_interface=127.18.96.190
--webserver_port=36951
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:34945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:17.272408 27450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:17.272670 27450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:17.272694 27450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:17.273897 27450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:17.273941 27450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:17.273957 27450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:17.273973 27450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:17.273985 27450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:17.275478 27450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:34945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:17.275755 27450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:17.276026 27450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:17.280896 27456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:17.280889 27458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:17.280889 27455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:17.281261 27450 server_base.cc:1034] running on GCE node
I20260928 14:48:17.281533 27450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:17.281841 27450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:17.283022 27450 hybrid_clock.cc:648] HybridClock initialized: now 1790606897282990 us; error 60 us; skew 500 ppm
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606897, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:17.285679 27450 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:48:17.286150 27450 webserver.cc:460] Webserver started at http://127.18.96.190:36951/ using document root <none> and password file <none>
I20260928 14:48:17.286341 27450 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:17.286394 27450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:17.287784 27450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:17.288532 27465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.288800 27450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:48:17.288894 27450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "7ec0fc3442984433ab2a0cad81b3deac"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "34f07c1e382ff37385ee7afc43d07248"
server_key_iv: "1e60a52f41d9a6a7ad763cc26333d66b"
server_key_version: "encryptionkey@0"
I20260928 14:48:17.289240 27450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:17.297384 27450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:17.299623 27450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:17.299734 27450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:17.303550 27450 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:34945
I20260928 14:48:17.303598 27517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:34945 every 8 connection(s)
I20260928 14:48:17.304011 27450 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:17.304703 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 27450
I20260928 14:48:17.305034 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:33575
--local_ip_for_outbound_sockets=127.18.96.129
--tserver_master_addrs=127.18.96.190:34945
--webserver_port=37281
--webserver_interface=127.18.96.129
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.96.129
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:17.306150 27518 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:48:17.306970 27518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: Bootstrap starting.
I20260928 14:48:17.315995 27518 log.cc:824] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:17.317721 27518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:48:17.318035 27518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: Bootstrap complete.
I20260928 14:48:17.320478 27518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } }
I20260928 14:48:17.320703 27518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec0fc3442984433ab2a0cad81b3deac, State: Initialized, Role: FOLLOWER
I20260928 14:48:17.320869 27518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } }
I20260928 14:48:17.320942 27518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:17.320990 27518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:17.321038 27518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:48:17.322069 27518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } }
I20260928 14:48:17.322155 27518 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec0fc3442984433ab2a0cad81b3deac; no voters:
I20260928 14:48:17.322357 27518 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:48:17.322407 27522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:48:17.322531 27522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [term 2 LEADER]: Becoming Leader. State: Replica: 7ec0fc3442984433ab2a0cad81b3deac, State: Running, Role: LEADER
I20260928 14:48:17.322635 27518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:17.322618 27522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } }
I20260928 14:48:17.322916 27522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec0fc3442984433ab2a0cad81b3deac. Latest consensus state: current_term: 2 leader_uuid: "7ec0fc3442984433ab2a0cad81b3deac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } } }
I20260928 14:48:17.322992 27522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:17.322921 27524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7ec0fc3442984433ab2a0cad81b3deac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec0fc3442984433ab2a0cad81b3deac" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34945 } } }
I20260928 14:48:17.323190 27524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:17.323324 27526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:17.323904 27526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:17.324045 27526 catalog_manager.cc:1259] Loaded cluster ID: fcd59823ca084c07b3cb7578cca85c16
I20260928 14:48:17.324084 27526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:17.325237 27526 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:17.325466 27526 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ec0fc3442984433ab2a0cad81b3deac: Loaded TSK: 0
I20260928 14:48:17.325700 27526 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:48:17.390290 27520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:17.390529 27520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:17.390571 27520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:17.391840 27520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:17.391923 27520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:17.391970 27520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:17.393565 27520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.129.keytab
--principal=oryx/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:33575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=37281
--tserver_master_addrs=127.18.96.190:34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:17.393831 27520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:17.394069 27520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:17.399935 27547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:17.399942 27545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:17.400043 27544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:17.400273 27520 server_base.cc:1034] running on GCE node
I20260928 14:48:17.400560 27520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:17.400768 27520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:17.401939 27520 hybrid_clock.cc:648] HybridClock initialized: now 1790606897401917 us; error 38 us; skew 500 ppm
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606897, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:17.404491 27520 init.cc:377] Logged in from keytab as oryx/127.18.96.129@KRBTEST.COM (short username oryx)
I20260928 14:48:17.405016 27520 webserver.cc:460] Webserver started at http://127.18.96.129:37281/ using document root <none> and password file <none>
I20260928 14:48:17.405254 27520 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:17.405311 27520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:17.406962 27520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:17.407649 27554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.407821 27520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:17.407941 27520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "8ce62bf5ce9e43cebb485643ce247013"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "1274a1d12d6695e97f358d8c37eab9c1"
server_key_iv: "7e01c4cf54fc3ab90782ec7ac0dc08d3"
server_key_version: "encryptionkey@0"
I20260928 14:48:17.408327 27520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:17.415508 27520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:17.423777 27520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:17.423991 27520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:17.424257 27520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:17.424640 27520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:17.424678 27520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.424700 27520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:17.424712 27520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.431967 27520 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:33575
I20260928 14:48:17.432013 27667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:33575 every 8 connection(s)
I20260928 14:48:17.432363 27520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:17.440467 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 27520
I20260928 14:48:17.440744 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:37087
--local_ip_for_outbound_sockets=127.18.96.130
--tserver_master_addrs=127.18.96.190:34945
--webserver_port=46255
--webserver_interface=127.18.96.130
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.96.130
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:17.441268 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.433028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:55475)
0928 14:48:17.433336 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.433385 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.433950 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.434490 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.434495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.435096 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.435296 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.435301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.436511 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.436512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.437323 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.437325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.437927 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.437928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.438060 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.438597 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.438605 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.441077 (+ 2472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":86,"threads_started":1}
W20260928 14:48:17.441488 27668 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:34945 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.96.190:34945: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
W20260928 14:48:17.441802 27671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:17.433161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:55475 (local address 127.18.96.190:34945)
0928 14:48:17.434130 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.434137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.434161 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:48:17.434255 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.434259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.434298 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.434436 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.435422 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.436405 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.437417 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.437821 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.437950 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.441239 (+ 3289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:55475: BlockingRecv error: recv got EOF from 127.18.96.129:55475 (error 108)
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":86,"threads_started":1}
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:17.446417 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.441677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:44447)
0928 14:48:17.441988 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.441999 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.442040 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.442286 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.442287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.442421 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.442469 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.442470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.443127 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.443128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.443628 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.443629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.444066 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.444067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.444143 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.444598 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.444603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.446330 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":79,"threads_started":1}
W20260928 14:48:17.446467 27671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:17.441838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:44447 (local address 127.18.96.190:34945)
0928 14:48:17.442124 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.442127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.442137 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:17.442162 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.442163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.442176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.442275 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.442558 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.443022 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.443721 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.443977 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.444064 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.446405 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:44447: BlockingRecv error: recv got EOF from 127.18.96.129:44447 (error 108)
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":98,"threads_started":1}
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:17.450984 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.446715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:49791)
0928 14:48:17.446825 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.446836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.446880 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.447041 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.447042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.447170 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.447196 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.447197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.447819 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.447820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.448419 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.448425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.448828 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.448829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.448894 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.449333 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.449338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.450906 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 14:48:17.451031 27671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:17.446816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:49791 (local address 127.18.96.190:34945)
0928 14:48:17.446852 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.446854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.446864 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:17.446954 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.446955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.446964 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.447035 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.447279 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.447736 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.448540 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.448734 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.448827 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.450970 (+ 2143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:49791: BlockingRecv error: recv got EOF from 127.18.96.129:49791 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 14:48:17.451331 27668 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:48:17.529732 27674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:17.529944 27674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:17.529969 27674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:17.531075 27674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:17.531111 27674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:17.531136 27674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:17.532523 27674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.130.keytab
--principal=oryx/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:37087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=46255
--tserver_master_addrs=127.18.96.190:34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:17.532711 27674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:17.532883 27674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:17.538118 27679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:17.538139 27680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:17.538228 27674 server_base.cc:1034] running on GCE node
W20260928 14:48:17.538125 27682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:17.538552 27674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:17.538805 27674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:17.539969 27674 hybrid_clock.cc:648] HybridClock initialized: now 1790606897539954 us; error 37 us; skew 500 ppm
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606897, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:17.542398 27674 init.cc:377] Logged in from keytab as oryx/127.18.96.130@KRBTEST.COM (short username oryx)
I20260928 14:48:17.542870 27674 webserver.cc:460] Webserver started at http://127.18.96.130:46255/ using document root <none> and password file <none>
I20260928 14:48:17.543169 27674 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:17.543263 27674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:17.544637 27674 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:48:17.545258 27689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.545423 27674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:48:17.545521 27674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "00253214a854481a9e525b530bf2fe05"
format_stamp: "Formatted at 2026-09-28 14:48:16 on dist-test-slave-j968"
server_key: "1382970318fc582fe612d76d611f1cb2"
server_key_iv: "da93e1d11ea1e705e449f8adace19dc6"
server_key_version: "encryptionkey@0"
I20260928 14:48:17.545851 27674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:17.566638 27674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:17.569029 27674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:17.569203 27674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:17.569446 27674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:17.569756 27674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:17.569792 27674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.569844 27674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:17.569875 27674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.575929 27674 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:37087
I20260928 14:48:17.576015 27802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:37087 every 8 connection(s)
I20260928 14:48:17.576376 27674 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:17.576601 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 27674
I20260928 14:48:17.576949 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:37005
--local_ip_for_outbound_sockets=127.18.96.131
--tserver_master_addrs=127.18.96.190:34945
--webserver_port=36041
--webserver_interface=127.18.96.131
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.96.131
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:48:17.584437 27671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:17.577585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:57121 (local address 127.18.96.190:34945)
0928 14:48:17.577634 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.577636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.577647 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:48:17.578157 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.578158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.578181 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.578244 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.579185 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.579685 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.580761 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.580986 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.581065 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.584379 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:57121: BlockingRecv error: recv got EOF from 127.18.96.130:57121 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:48:17.584502 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.577265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:57121)
0928 14:48:17.577436 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.577499 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.578040 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.578320 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.578324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.578878 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.579074 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.579078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.579777 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.579778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.580656 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.580659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.581069 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.581070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.581199 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.581816 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.581825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.584304 (+ 2479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
W20260928 14:48:17.584715 27803 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:34945 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.96.190:34945: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:17.589578 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.584972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:55401)
0928 14:48:17.585028 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.585039 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.585078 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.585213 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.585214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.585373 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.585434 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.585436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.586036 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.586037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.586984 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.586987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.587316 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.587317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.587384 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.587763 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.587768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.589511 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:48:17.589753 27671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:17.585020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:55401 (local address 127.18.96.190:34945)
0928 14:48:17.585085 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.585087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.585097 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:17.585133 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.585134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.585141 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.585191 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.585515 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.585958 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.587069 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.587244 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.587295 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.589665 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:55401: BlockingRecv error: recv got EOF from 127.18.96.130:55401 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:17.594590 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.589791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:34437)
0928 14:48:17.589897 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.589906 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.589939 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.590155 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.590156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.590314 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.590345 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.590346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.590949 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.590950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.591773 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.591775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.592312 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.592312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.592368 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.592765 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.592770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.594510 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
W20260928 14:48:17.594640 27671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:17.589983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:34437 (local address 127.18.96.190:34945)
0928 14:48:17.590022 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.590024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.590034 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:17.590057 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.590058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.590067 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.590131 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.590422 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.590873 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.591859 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.592220 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.592286 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.594576 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:34437: BlockingRecv error: recv got EOF from 127.18.96.130:34437 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 14:48:17.594725 27803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:48:17.663385 27806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:17.663596 27806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:17.663619 27806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:17.664831 27806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:17.664877 27806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:17.664903 27806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:17.666323 27806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/krb5kdc/oryx_127.18.96.131.keytab
--principal=oryx/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:37005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=36041
--tserver_master_addrs=127.18.96.190:34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:17.666563 27806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:17.666817 27806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:17.671917 27813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:17.671900 27815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:17.671895 27812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:17.671948 27806 server_base.cc:1034] running on GCE node
I20260928 14:48:17.672339 27806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:17.672600 27806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:17.673744 27806 hybrid_clock.cc:648] HybridClock initialized: now 1790606897673737 us; error 28 us; skew 500 ppm
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606897, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:17.676388 27806 init.cc:377] Logged in from keytab as oryx/127.18.96.131@KRBTEST.COM (short username oryx)
I20260928 14:48:17.676822 27806 webserver.cc:460] Webserver started at http://127.18.96.131:36041/ using document root <none> and password file <none>
I20260928 14:48:17.677060 27806 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:17.677116 27806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:17.678481 27806 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:17.679096 27822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.679276 27806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:17.679376 27806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "fcc480cab46b4937910fef077db87735"
format_stamp: "Formatted at 2026-09-28 14:48:17 on dist-test-slave-j968"
server_key: "0ca79ce1214bf34a7b49275d12db4e68"
server_key_iv: "fa8259b9f1372907a545caef7eb84bf7"
server_key_version: "encryptionkey@0"
I20260928 14:48:17.679733 27806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:17.707623 27806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:17.710047 27806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:17.710230 27806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:17.710501 27806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:17.710821 27806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:17.710876 27806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.710919 27806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:17.710947 27806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:17.716778 27806 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:37005
I20260928 14:48:17.716842 27935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:37005 every 8 connection(s)
I20260928 14:48:17.717198 27806 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:17.722277 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 27806
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:48:17.724920 27671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:17.717943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:42661 (local address 127.18.96.190:34945)
0928 14:48:17.717991 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.717993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.718196 (+ 203us) server_negotiation.cc:407] Connection header received
0928 14:48:17.718659 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.718660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.718671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.718727 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.719617 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.720128 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.721147 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.721330 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.721381 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.724861 (+ 3480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:42661: BlockingRecv error: recv got EOF from 127.18.96.131:42661 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:48:17.725006 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.717917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:42661)
0928 14:48:17.718061 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.718095 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.718571 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.718779 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.718782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.719335 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.719535 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.719539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.720205 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.720206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.721056 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.721058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.721399 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.721400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.721530 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.722118 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.722134 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.724800 (+ 2666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
W20260928 14:48:17.725188 27936 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:34945 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.96.190:34945: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:17.729978 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.725435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:34651)
0928 14:48:17.725475 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.725483 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.725537 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.725665 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.725666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.725791 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.725834 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.725835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.726673 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.726674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.727198 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.727200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.727704 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.727706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.727784 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.728133 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.728138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.729897 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 14:48:17.730026 27671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:17.725497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:34651 (local address 127.18.96.190:34945)
0928 14:48:17.725555 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.725557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.725565 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:48:17.725589 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.725590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.725598 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.725661 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.725909 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.726559 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.727296 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.727601 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.727681 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.729964 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:34651: BlockingRecv error: recv got EOF from 127.18.96.131:34651 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:48:17.730877 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.722802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.18.96.190:34945)
0928 14:48:17.722888 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.722891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.722959 (+ 68us) server_negotiation.cc:407] Connection header received
0928 14:48:17.723042 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.723042 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.723053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.723188 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.723983 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.724524 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.725532 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.725742 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.725773 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.727169 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:17.727174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:17.727174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:17.727193 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:17.729195 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:17.729581 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:17.729584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:17.729585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:17.729733 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:17.729951 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:17.729952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:17.729953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:17.730260 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:17.730398 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:17.730651 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 14:48:17.730767 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:17 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:17.734484 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:17.730306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:52163)
0928 14:48:17.730348 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:17.730357 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:17.730394 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:17.730545 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:17.730546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:17.730673 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:17.730697 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.730699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.731445 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.731446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.732150 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:17.732152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.732506 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.732507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:17.732560 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.732901 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:17.732905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:17.734414 (+ 1509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 14:48:17.734575 27671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:17.730350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:52163 (local address 127.18.96.190:34945)
0928 14:48:17.730389 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:17.730391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:17.730398 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:48:17.730452 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:17.730453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:17.730469 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:17.730542 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:17.730771 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.731365 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.732230 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:17.732425 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:17.732506 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:17.734509 (+ 2003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:52163: BlockingRecv error: recv got EOF from 127.18.96.131:52163 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:48:17.734642 27936 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 14:48:18 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:18 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:18.456641 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:18.451750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:38795)
0928 14:48:18.451935 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:18.451951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:18.452010 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:18.452262 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:18.452263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:18.452513 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:18.452592 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:18.452596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.453337 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.453338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:18.453914 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:18.453916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.454278 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.454279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:18.454319 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:18.454778 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:18.454782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:18.456552 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
W20260928 14:48:18.456696 27941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:18.451859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:38795 (local address 127.18.96.190:34945)
0928 14:48:18.452090 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:18.452092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:18.452102 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:18.452133 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:18.452134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:18.452147 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:18.452240 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:18.452676 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.453261 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.454019 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.454211 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.454264 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:18.456643 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:38795: BlockingRecv error: recv got EOF from 127.18.96.129:38795 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":130,"threads_started":1}
Sep 28 14:48:18 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:18 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:18.599711 27942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:18.595090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:41717)
0928 14:48:18.595227 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:18.595237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:18.595293 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:18.595429 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:18.595430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:18.595588 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:18.595636 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:18.595638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.596303 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.596304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:18.596972 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:18.596975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.597301 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.597302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:18.597343 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:18.597853 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:18.597858 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:18.599643 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
W20260928 14:48:18.599826 27941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:18.595164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:41717 (local address 127.18.96.190:34945)
0928 14:48:18.595224 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:18.595228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:18.595291 (+ 63us) server_negotiation.cc:407] Connection header received
0928 14:48:18.595335 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:18.595336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:18.595346 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:18.595397 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:18.595695 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.596228 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.597051 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.597238 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.597289 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:18.599746 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:41717: BlockingRecv error: recv got EOF from 127.18.96.130:41717 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 14:48:18 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:18 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:18.740301 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:18.735230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:40815)
0928 14:48:18.735386 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:18.735402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:18.735480 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:18.735636 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:18.735637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:18.735802 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:18.735853 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:18.735855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.736747 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.736748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:18.737323 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:18.737325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.737683 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.737683 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:18.737738 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:18.738240 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:18.738245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:18.740194 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
W20260928 14:48:18.740352 27941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:18.735318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:40815 (local address 127.18.96.190:34945)
0928 14:48:18.735409 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:18.735411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:18.735487 (+ 76us) server_negotiation.cc:407] Connection header received
0928 14:48:18.735527 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:18.735528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:18.735541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:18.735619 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:18.736064 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.736659 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.737415 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:18.737607 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:18.737684 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:18.740286 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:40815: BlockingRecv error: recv got EOF from 127.18.96.131:40815 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 14:48:19 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:19 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:19.462107 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:19.457186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:60243)
0928 14:48:19.457363 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:19.457374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:19.457424 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:19.457574 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:19.457574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:19.457755 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:19.457808 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:19.457810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.458490 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.458490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:19.459055 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:19.459057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.459472 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.459474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:19.459526 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:19.460111 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:19.460116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:19.462050 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
W20260928 14:48:19.462285 27945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:19.457260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:60243 (local address 127.18.96.190:34945)
0928 14:48:19.457393 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:19.457395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:19.457408 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:48:19.457472 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:19.457473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:19.457486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:19.457552 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:19.457880 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.458413 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.459154 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.459361 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.459427 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:19.462193 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:60243: BlockingRecv error: recv got EOF from 127.18.96.129:60243 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
Sep 28 14:48:19 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:19 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:19.605429 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:19.600282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:56403)
0928 14:48:19.600453 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:19.600466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:19.600536 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:19.600671 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:19.600672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:19.600824 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:19.600876 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:19.600878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.601597 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.601598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:19.602156 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:19.602158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.602672 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.602673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:19.602735 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:19.603252 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:19.603257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:19.605332 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
W20260928 14:48:19.605528 27945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:19.600411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:56403 (local address 127.18.96.190:34945)
0928 14:48:19.600483 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:19.600485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:19.600533 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:48:19.600589 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:19.600590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:19.600600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:19.600652 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:19.601018 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.601518 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.602267 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.602599 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.602674 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:19.605428 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:56403: BlockingRecv error: recv got EOF from 127.18.96.130:56403 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 14:48:19 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:19 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:19.745869 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:19.740887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:47677)
0928 14:48:19.741064 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:19.741078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:19.741153 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:19.741386 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:19.741387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:19.741584 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:19.741632 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:19.741634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.742337 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.742338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:19.742869 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:19.742870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.743251 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.743252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:19.743313 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:19.743825 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:19.743830 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:19.745809 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":58,"threads_started":1}
W20260928 14:48:19.746052 27945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:19.741181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:47677 (local address 127.18.96.190:34945)
0928 14:48:19.741253 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:19.741255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:19.741268 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:48:19.741292 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:19.741293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:19.741303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:19.741379 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:19.741725 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.742249 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.742959 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:19.743165 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:19.743241 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:19.745983 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:47677: BlockingRecv error: recv got EOF from 127.18.96.131:47677 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:48:20 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:20 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:20.467196 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:20.462628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:42447)
0928 14:48:20.462777 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:20.462794 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:20.462847 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:20.462999 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:20.463000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:20.463166 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:20.463217 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:20.463218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.463910 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.463912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:20.464468 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:20.464470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.464806 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.464807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:20.464845 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:20.465284 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:20.465289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:20.467118 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
W20260928 14:48:20.467271 27949 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:20.462689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:42447 (local address 127.18.96.190:34945)
0928 14:48:20.462833 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:20.462835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:20.462845 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:20.462900 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:20.462900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:20.462910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:20.462973 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:20.463287 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.463797 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.464542 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.464741 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.464794 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:20.467218 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:42447: BlockingRecv error: recv got EOF from 127.18.96.129:42447 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
Sep 28 14:48:20 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:20 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:20.611416 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:20.605997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:57775)
0928 14:48:20.606234 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:20.606245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:20.606311 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:20.606555 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:20.606557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:20.606728 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:20.606781 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:20.606783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.607682 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.607683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:20.608298 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:20.608300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.608709 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.608710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:20.608760 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:20.609280 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:20.609284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:20.611334 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":161,"threads_started":1}
W20260928 14:48:20.611542 27949 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:20.606156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:57775 (local address 127.18.96.190:34945)
0928 14:48:20.606235 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:20.606237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:20.606346 (+ 109us) server_negotiation.cc:407] Connection header received
0928 14:48:20.606380 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:20.606381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:20.606398 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:20.606463 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:20.606896 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.607582 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.608400 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.608623 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.608686 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:20.611437 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:57775: BlockingRecv error: recv got EOF from 127.18.96.130:57775 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:48:20 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:20 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:20.751247 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:20.746360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:35379)
0928 14:48:20.746544 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:20.746556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:20.746652 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:20.746827 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:20.746828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:20.747033 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:20.747081 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:20.747083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.747771 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.747772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:20.748451 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:20.748453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.748814 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.748814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:20.748868 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:20.749381 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:20.749386 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:20.751148 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":107,"threads_started":1}
W20260928 14:48:20.751291 27949 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:20.746466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:35379 (local address 127.18.96.190:34945)
0928 14:48:20.746544 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:20.746546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:20.746605 (+ 59us) server_negotiation.cc:407] Connection header received
0928 14:48:20.746730 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:20.746731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:20.746746 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:20.746821 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:20.747197 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.747678 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.748533 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:20.748728 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:20.748831 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:20.751226 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:35379: BlockingRecv error: recv got EOF from 127.18.96.131:35379 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 14:48:21 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:21 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:21.472782 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:21.467785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:48941)
0928 14:48:21.467981 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:21.467993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:21.468053 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:21.468189 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:21.468190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:21.468359 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:21.468412 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:21.468414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.469188 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.469189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:21.469864 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:21.469866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.470231 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.470232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:21.470271 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:21.470792 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:21.470798 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:21.472712 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":131,"threads_started":1}
W20260928 14:48:21.472857 27953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:21.467932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:48941 (local address 127.18.96.190:34945)
0928 14:48:21.468057 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:21.468059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:21.468069 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:21.468097 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:21.468097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:21.468110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:21.468171 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:21.468486 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.469109 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.469948 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.470158 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.470213 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:21.472803 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:48941: BlockingRecv error: recv got EOF from 127.18.96.129:48941 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
Sep 28 14:48:21 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:21 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:21.616813 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:21.612054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:40689)
0928 14:48:21.612191 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:21.612202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:21.612261 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:21.612400 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:21.612401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:21.612568 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:21.612615 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:21.612617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.613281 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.613282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:21.613831 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:21.613832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.614200 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.614202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:21.614252 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:21.614728 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:21.614733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:21.616750 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
W20260928 14:48:21.616978 27953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:21.612119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:40689 (local address 127.18.96.190:34945)
0928 14:48:21.612192 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:21.612194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:21.612266 (+ 72us) server_negotiation.cc:407] Connection header received
0928 14:48:21.612313 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:21.612313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:21.612324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:21.612366 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:21.612702 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.613195 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.613917 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.614105 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.614171 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:21.616903 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:40689: BlockingRecv error: recv got EOF from 127.18.96.130:40689 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 14:48:21 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:21 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:21.756961 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:21.751806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:60639)
0928 14:48:21.751956 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:21.751990 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:21.752069 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:21.752282 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:21.752284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:21.752464 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:21.752517 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:21.752519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.753254 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.753256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:21.753900 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:21.753902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.754334 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.754335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:21.754397 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:21.754867 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:21.754871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:21.756898 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":110,"threads_started":1}
W20260928 14:48:21.757155 27953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:21.751951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:60639 (local address 127.18.96.190:34945)
0928 14:48:21.752019 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:21.752021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:21.752049 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:48:21.752180 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:21.752182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:21.752203 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:21.752273 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:21.752618 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.753160 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.753995 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:21.754196 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:21.754332 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:21.757073 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:60639: BlockingRecv error: recv got EOF from 127.18.96.131:60639 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 14:48:22 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:22 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:22.483692 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:22.473384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:51335)
0928 14:48:22.473568 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:22.473579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:22.473671 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:22.473816 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:22.473816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:22.473972 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:22.474027 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:22.474028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.474687 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.474688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:22.475205 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:22.475206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.475585 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.475585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:22.475633 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:22.481327 (+ 5694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:22.481332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:22.483582 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
W20260928 14:48:22.483759 27957 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:22.473505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:51335 (local address 127.18.96.190:34945)
0928 14:48:22.473641 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:22.473643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:22.473656 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:48:22.473725 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:22.473726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:22.473736 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:22.473795 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:22.474101 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.474612 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.475299 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.475511 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.475573 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:22.483681 (+ 8108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:51335: BlockingRecv error: recv got EOF from 127.18.96.129:51335 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
Sep 28 14:48:22 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:22 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:22.622651 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:22.617308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:57795)
0928 14:48:22.617500 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:22.617515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:22.617587 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:22.617744 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:22.617745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:22.617991 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:22.618058 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:22.618061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.618752 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.618752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:22.619329 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:22.619331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.619713 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.619714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:22.619768 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:22.620486 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:22.620491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:22.622525 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
W20260928 14:48:22.622756 27957 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:22.617469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:57795 (local address 127.18.96.190:34945)
0928 14:48:22.617536 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:22.617537 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:22.617570 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:48:22.617660 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:22.617661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:22.617671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:22.617723 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:22.618146 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.618671 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.619404 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.619603 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.619679 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:22.622637 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:57795: BlockingRecv error: recv got EOF from 127.18.96.130:57795 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 14:48:22 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:22 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:22.762327 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:22.757535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:33675)
0928 14:48:22.757698 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:22.757712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:22.757775 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:22.757946 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:22.757948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:22.758128 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:22.758181 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:22.758183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.758929 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.758931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:22.759486 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:22.759488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.759827 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.759827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:22.759896 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:22.760387 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:22.760391 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:22.762267 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
W20260928 14:48:22.762512 27957 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:22.757613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:33675 (local address 127.18.96.190:34945)
0928 14:48:22.757703 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:22.757709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:22.757778 (+ 69us) server_negotiation.cc:407] Connection header received
0928 14:48:22.757823 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:22.757824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:22.757836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:22.757894 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:22.758295 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.758830 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.759572 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:22.759756 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:22.759811 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:22.762435 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:33675: BlockingRecv error: recv got EOF from 127.18.96.131:33675 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 14:48:23 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:23 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:23.489418 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:23.484208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:35415)
0928 14:48:23.484372 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:23.484382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:23.484440 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:23.484576 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:23.484576 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:23.484741 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:23.484792 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:23.484794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.485480 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.485481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:23.486384 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:23.486388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.486852 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.486853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:23.486897 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:23.487369 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:23.487374 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:23.489299 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
W20260928 14:48:23.489455 27961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:23.484307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:35415 (local address 127.18.96.190:34945)
0928 14:48:23.484448 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:23.484450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:23.484459 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:23.484486 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:23.484487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:23.484497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:23.484560 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:23.484867 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.485405 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.486528 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.486783 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.486842 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:23.489382 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:35415: BlockingRecv error: recv got EOF from 127.18.96.129:35415 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":55,"threads_started":1}
Sep 28 14:48:23 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:23 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:23.628266 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:23.623352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:56163)
0928 14:48:23.623504 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:23.623514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:23.623564 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:23.623717 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:23.623718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:23.623938 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:23.623985 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:23.623987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.624681 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.624682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:23.625190 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:23.625192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.625528 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.625529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:23.625567 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:23.626073 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:23.626077 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:23.628166 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
W20260928 14:48:23.628304 27961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:23.623407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:56163 (local address 127.18.96.190:34945)
0928 14:48:23.623538 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:23.623542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:23.623559 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:48:23.623634 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:23.623635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:23.623647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:23.623711 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:23.624057 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.624613 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.625258 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.625464 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.625519 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:23.628245 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:56163: BlockingRecv error: recv got EOF from 127.18.96.130:56163 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 14:48:23 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:23 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:23.767763 27963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:23.762899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:46977)
0928 14:48:23.763070 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:23.763090 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:23.763159 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:23.763333 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:23.763333 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:23.763501 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:23.763551 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:23.763553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.764272 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.764273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:23.764832 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:23.764834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.765201 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.765202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:23.765245 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:23.765698 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:23.765703 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:23.767635 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
W20260928 14:48:23.767781 27961 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:23.763122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:46977 (local address 127.18.96.190:34945)
0928 14:48:23.763194 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:23.763196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:23.763206 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:23.763241 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:23.763242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:23.763253 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:23.763305 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:23.763625 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.764181 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.764931 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:23.765127 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:23.765187 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:23.767725 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:46977: BlockingRecv error: recv got EOF from 127.18.96.131:46977 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:48:24 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:24 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:24.495054 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:24.489961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:54529)
0928 14:48:24.490159 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:24.490172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:24.490239 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:24.490530 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:24.490531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:24.490784 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:24.490844 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:24.490846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.491556 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.491557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:24.492129 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:24.492131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.492540 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.492541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:24.492580 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:24.493104 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:24.493110 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:24.494994 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
W20260928 14:48:24.495219 27965 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:24.490079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:54529 (local address 127.18.96.190:34945)
0928 14:48:24.490316 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:24.490318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:24.490328 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:24.490360 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:24.490361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:24.490371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:24.490507 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:24.490952 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.491483 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.492202 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.492409 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.492525 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:24.495101 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:54529: BlockingRecv error: recv got EOF from 127.18.96.129:54529 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
Sep 28 14:48:24 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:24 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:24.633728 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:24.628727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:53747)
0928 14:48:24.628916 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:24.628928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:24.629024 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:24.629186 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:24.629187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:24.629373 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:24.629429 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:24.629431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.630160 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.630162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:24.630724 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:24.630725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.631111 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.631112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:24.631171 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:24.631593 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:24.631598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:24.633669 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
W20260928 14:48:24.633872 27965 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:24.628824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:53747 (local address 127.18.96.190:34945)
0928 14:48:24.628900 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:24.628903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:24.629007 (+ 104us) server_negotiation.cc:407] Connection header received
0928 14:48:24.629095 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:24.629096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:24.629114 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:24.629179 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:24.629516 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.630067 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.630816 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.631023 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.631110 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:24.633793 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:53747: BlockingRecv error: recv got EOF from 127.18.96.130:53747 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:48:24 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:24 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:24.773465 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:24.768386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:42303)
0928 14:48:24.768639 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:24.768651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:24.768720 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:24.768882 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:24.768883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:24.769045 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:24.769099 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:24.769101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.769858 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.769858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:24.770420 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:24.770422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.770799 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.770800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:24.770842 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:24.771312 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:24.771316 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:24.773407 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":106,"threads_started":1}
W20260928 14:48:24.773612 27965 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:24.768475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:42303 (local address 127.18.96.190:34945)
0928 14:48:24.768553 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:24.768556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:24.768753 (+ 197us) server_negotiation.cc:407] Connection header received
0928 14:48:24.768785 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:24.768785 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:24.768800 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:24.768861 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:24.769201 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.769779 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.770520 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:24.770724 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:24.770789 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:24.773529 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:42303: BlockingRecv error: recv got EOF from 127.18.96.131:42303 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 14:48:25 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:25 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:25.500525 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:25.495615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:44517)
0928 14:48:25.495754 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:25.495770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:25.495837 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:25.496083 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:25.496084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:25.496266 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:25.496321 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:25.496323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.496967 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.496968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:25.497585 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:25.497587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.497942 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.497944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:25.497998 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:25.498533 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:25.498538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:25.500412 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
W20260928 14:48:25.500573 27969 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:25.495726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:44517 (local address 127.18.96.190:34945)
0928 14:48:25.495940 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:25.495942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:25.495951 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:25.495975 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:25.495976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:25.495989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:25.496054 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:25.496392 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.496888 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.497659 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.497853 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.497904 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:25.500500 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:44517: BlockingRecv error: recv got EOF from 127.18.96.129:44517 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
Sep 28 14:48:25 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:25 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:25.639525 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:25.634260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:53881)
0928 14:48:25.634489 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:25.634502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:25.634601 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:25.634742 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:25.634743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:25.635004 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:25.635054 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:25.635056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.635936 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.635937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:25.636474 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:25.636476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.636854 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.636856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:25.636934 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:25.637521 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:25.637534 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:25.639467 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":110,"threads_started":1}
W20260928 14:48:25.639667 27969 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:25.634364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:53881 (local address 127.18.96.190:34945)
0928 14:48:25.634474 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:25.634477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:25.634584 (+ 107us) server_negotiation.cc:407] Connection header received
0928 14:48:25.634664 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:25.634665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:25.634679 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:25.634727 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:25.635148 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.635813 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.636565 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.636766 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.636851 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:25.639581 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:53881: BlockingRecv error: recv got EOF from 127.18.96.130:53881 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 14:48:25 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:25 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:25.778465 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:25.773951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:46387)
0928 14:48:25.774148 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:25.774161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:25.774224 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:25.774374 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:25.774374 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:25.774546 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:25.774596 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:25.774597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.775252 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.775253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:25.775796 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:25.775797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.776128 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.776129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:25.776170 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:25.776617 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:25.776621 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:25.778394 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
W20260928 14:48:25.778623 27969 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:25.774100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:46387 (local address 127.18.96.190:34945)
0928 14:48:25.774187 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:25.774188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:25.774201 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:48:25.774284 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:25.774285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:25.774299 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:25.774354 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:25.774667 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.775178 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.775887 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:25.776067 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:25.776117 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:25.778520 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:46387: BlockingRecv error: recv got EOF from 127.18.96.131:46387 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 14:48:26 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:26 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:26.506134 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:26.501121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:48653)
0928 14:48:26.501262 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:26.501279 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:26.501330 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:26.501533 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:26.501535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:26.501716 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:26.501770 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:26.501772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.502482 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.502483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:26.503062 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:26.503063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.503437 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.503439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:26.503507 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:26.504297 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:26.504302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:26.506077 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
W20260928 14:48:26.506310 27973 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:26.501242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:48653 (local address 127.18.96.190:34945)
0928 14:48:26.501372 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:26.501374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:26.501382 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:48:26.501405 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:26.501406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:26.501416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:26.501484 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:26.501853 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.502403 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.503156 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.503350 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.503407 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:26.506204 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:48653: BlockingRecv error: recv got EOF from 127.18.96.129:48653 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
Sep 28 14:48:26 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:26 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:26.644838 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:26.640056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:33045)
0928 14:48:26.640218 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:26.640228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:26.640285 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:26.640431 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:26.640432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:26.640642 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:26.640689 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:26.640691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.641325 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.641326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:26.641860 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:26.641861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.642242 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.642243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:26.642288 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:26.642778 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:26.642782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:26.644768 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
W20260928 14:48:26.645076 27973 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:26.640151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:33045 (local address 127.18.96.190:34945)
0928 14:48:26.640225 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:26.640230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:26.640272 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:48:26.640342 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:26.640343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:26.640355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:26.640398 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:26.640759 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.641252 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.641969 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.642167 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.642230 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:26.645015 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:33045: BlockingRecv error: recv got EOF from 127.18.96.130:33045 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 14:48:26 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:26 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:26.783967 27975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:26.779030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:37071)
0928 14:48:26.779222 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:26.779232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:26.779323 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:26.779469 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:26.779469 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:26.779637 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:26.779696 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:26.779698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.780484 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.780485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:26.781021 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:26.781023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.781372 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.781373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:26.781410 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:26.781925 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:26.781930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:26.783843 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
W20260928 14:48:26.784018 27973 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:26.779077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:37071 (local address 127.18.96.190:34945)
0928 14:48:26.779129 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:26.779131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:26.779300 (+ 169us) server_negotiation.cc:407] Connection header received
0928 14:48:26.779386 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:26.779387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:26.779402 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:26.779445 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:26.779793 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.780395 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.781118 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:26.781302 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:26.781365 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:26.783954 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:37071: BlockingRecv error: recv got EOF from 127.18.96.131:37071 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 14:48:27 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:27 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:27.518280 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:27.506636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:34927)
0928 14:48:27.506813 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:27.506825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:27.506901 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:27.507088 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:27.507089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:27.507265 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:27.507314 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:27.507316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.508097 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.508099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:27.508688 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:27.508690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.509112 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.509113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:27.509162 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:27.515985 (+ 6823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:27.515991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:27.518211 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
W20260928 14:48:27.518424 27977 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:27.506713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:34927 (local address 127.18.96.190:34945)
0928 14:48:27.506820 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:27.506822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:27.506881 (+ 59us) server_negotiation.cc:407] Connection header received
0928 14:48:27.506977 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:27.506978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:27.506988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:27.507046 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:27.507392 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.508001 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.508763 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.508951 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.509002 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:27.518313 (+ 9311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:34927: BlockingRecv error: recv got EOF from 127.18.96.129:34927 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":49,"threads_started":1}
Sep 28 14:48:27 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:27 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:27.650521 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:27.645350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:35507)
0928 14:48:27.645549 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:27.645561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:27.645635 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:27.645781 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:27.645782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:27.645938 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:27.645988 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:27.645990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.646646 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.646647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:27.647193 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:27.647195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.647597 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.647598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:27.647664 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:27.648272 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:27.648278 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:27.650425 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
W20260928 14:48:27.650602 27977 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:27.645479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:35507 (local address 127.18.96.190:34945)
0928 14:48:27.645540 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:27.645542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:27.645672 (+ 130us) server_negotiation.cc:407] Connection header received
0928 14:48:27.645702 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:27.645702 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:27.645714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:27.645754 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:27.646052 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.646547 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.647274 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.647497 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.647572 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:27.650554 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:35507: BlockingRecv error: recv got EOF from 127.18.96.130:35507 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 14:48:27 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:27 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:27.788982 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:27.784464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:37889)
0928 14:48:27.784614 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:27.784627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:27.784703 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:27.784842 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:27.784843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:27.785005 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:27.785056 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:27.785058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.785684 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.785684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:27.786227 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:27.786228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.786542 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.786542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:27.786579 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:27.787039 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:27.787043 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:27.788915 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
W20260928 14:48:27.789101 27977 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:27.784601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:37889 (local address 127.18.96.190:34945)
0928 14:48:27.784692 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:27.784694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:27.784703 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:27.784758 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:27.784759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:27.784771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:27.784820 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:27.785123 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.785612 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.786301 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:27.786479 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:27.786528 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:27.789020 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:37889: BlockingRecv error: recv got EOF from 127.18.96.131:37889 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 14:48:28 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:28 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:28.524330 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:28.518984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:44989)
0928 14:48:28.519118 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:28.519131 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:28.519194 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:28.519387 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:28.519388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:28.519559 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:28.519611 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:28.519612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.520330 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.520331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:28.520884 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:28.520886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.521290 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.521291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:28.521342 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:28.521940 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:28.521945 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:28.524219 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
W20260928 14:48:28.524377 27981 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:28.519070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:44989 (local address 127.18.96.190:34945)
0928 14:48:28.519212 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:28.519214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:28.519224 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:28.519253 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:28.519254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:28.519269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:28.519334 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:28.519690 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.520261 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.521005 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.521197 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.521248 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:28.524303 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:44989: BlockingRecv error: recv got EOF from 127.18.96.129:44989 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
Sep 28 14:48:28 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:28 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:28.656358 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:28.651139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:54813)
0928 14:48:28.651317 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:28.651338 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:28.651409 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:28.651569 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:28.651570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:28.651752 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:28.651805 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:28.651806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.652559 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.652559 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:28.653257 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:28.653260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.653678 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.653679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:28.653730 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:28.654218 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:28.654223 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:28.656282 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
W20260928 14:48:28.656424 27981 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:28.651243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:54813 (local address 127.18.96.190:34945)
0928 14:48:28.651314 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:28.651315 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:28.651409 (+ 94us) server_negotiation.cc:407] Connection header received
0928 14:48:28.651483 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:28.651484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:28.651495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:28.651540 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:28.651937 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.652475 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.653377 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.653582 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.653656 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:28.656377 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:54813: BlockingRecv error: recv got EOF from 127.18.96.130:54813 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 14:48:28 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:28 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:28.794305 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:28.789508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:40035)
0928 14:48:28.789653 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:28.789664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:28.789731 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:28.789898 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:28.789900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:28.790148 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:28.790194 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:28.790196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.790876 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.790877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:28.791426 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:28.791428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.791780 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.791781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:28.791844 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:28.792380 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:28.792385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:28.794198 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
W20260928 14:48:28.794334 27981 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:28.789694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:40035 (local address 127.18.96.190:34945)
0928 14:48:28.789768 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:28.789770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:28.789779 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:28.789805 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:28.789805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:28.789824 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:28.789892 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:28.790288 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.790791 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.791513 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:28.791705 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:28.791782 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:28.794272 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:40035: BlockingRecv error: recv got EOF from 127.18.96.131:40035 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 14:48:29 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:29 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:29.529814 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:29.524841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:35049)
0928 14:48:29.525127 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:29.525142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:29.525231 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:29.525419 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:29.525420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:29.525590 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:29.525637 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:29.525639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.526402 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.526403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:29.526943 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:29.526944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.527318 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.527318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:29.527363 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:29.527996 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:29.528000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:29.529756 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":187,"threads_started":1}
W20260928 14:48:29.529955 27985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:29.525066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:35049 (local address 127.18.96.190:34945)
0928 14:48:29.525222 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:29.525224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:29.525234 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:29.525321 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:29.525322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:29.525333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:29.525394 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:29.525739 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.526315 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.527031 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.527242 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.527300 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:29.529860 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:35049: BlockingRecv error: recv got EOF from 127.18.96.129:35049 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
Sep 28 14:48:29 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:29 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:29.661461 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:29.656852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:40803)
0928 14:48:29.656997 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:29.657011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:29.657083 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:29.657251 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:29.657252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:29.657434 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:29.657482 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:29.657484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.658183 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.658184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:29.658711 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:29.658713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.659097 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.659098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:29.659144 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:29.659609 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:29.659613 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:29.661347 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
W20260928 14:48:29.661561 27985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:29.657042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:40803 (local address 127.18.96.190:34945)
0928 14:48:29.657133 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:29.657135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:29.657143 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:48:29.657169 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:29.657170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:29.657180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:29.657235 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:29.657552 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.658104 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.658787 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.659026 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.659084 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:29.661480 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:40803: BlockingRecv error: recv got EOF from 127.18.96.130:40803 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 14:48:29 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:29 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:29.799263 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:29.794807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:44803)
0928 14:48:29.794948 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:29.794963 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:29.795021 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:29.795141 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:29.795142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:29.795310 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:29.795358 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:29.795360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.796010 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.796011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:29.796533 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:29.796534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.796864 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.796865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:29.796959 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:29.797403 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:29.797407 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:29.799169 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
W20260928 14:48:29.799304 27985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:29.794867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:44803 (local address 127.18.96.190:34945)
0928 14:48:29.794934 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:29.794936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:29.795026 (+ 90us) server_negotiation.cc:407] Connection header received
0928 14:48:29.795064 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:29.795065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:29.795077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:29.795125 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:29.795416 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.795926 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.796608 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:29.796795 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:29.796849 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:29.799243 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:44803: BlockingRecv error: recv got EOF from 127.18.96.131:44803 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 14:48:30 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:30 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:30.535346 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:30.530375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:50579)
0928 14:48:30.530539 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:30.530549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:30.530612 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:30.530767 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:30.530767 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:30.530965 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:30.531018 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:30.531020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.531662 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.531663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:30.532267 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:30.532270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.532635 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.532636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:30.532687 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:30.533194 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:30.533199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:30.535239 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
W20260928 14:48:30.535406 27989 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:30.530503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:50579 (local address 127.18.96.190:34945)
0928 14:48:30.530642 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:30.530643 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:30.530652 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:48:30.530676 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:30.530677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:30.530688 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:30.530748 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:30.531090 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.531589 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.532349 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.532544 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.532601 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:30.535332 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:50579: BlockingRecv error: recv got EOF from 127.18.96.129:50579 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
Sep 28 14:48:30 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:30 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:30.667196 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:30.662016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:46543)
0928 14:48:30.662632 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:30.662645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:30.662718 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:30.662858 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:30.662859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:30.663086 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:30.663135 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:30.663136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.663806 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.663807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:30.664461 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:30.664463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.664809 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.664810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:30.664850 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:30.665336 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:30.665341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:30.667086 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":60,"threads_started":1}
W20260928 14:48:30.667244 27989 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:30.662186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:46543 (local address 127.18.96.190:34945)
0928 14:48:30.662258 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:30.662264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:30.662740 (+ 476us) server_negotiation.cc:407] Connection header received
0928 14:48:30.662772 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:30.662773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:30.662786 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:30.662841 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:30.663223 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.663728 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.664539 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.664741 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.664798 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:30.667180 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:46543: BlockingRecv error: recv got EOF from 127.18.96.130:46543 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 14:48:30 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:30 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:30.804886 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:30.799851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:48593)
0928 14:48:30.800125 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:30.800140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:30.800226 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:30.800382 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:30.800383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:30.800568 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:30.800614 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:30.800638 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.801335 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.801336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:30.801898 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:30.801900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.802271 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.802272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:30.802335 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:30.802838 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:30.802842 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:30.804780 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":210,"threads_started":1}
W20260928 14:48:30.804922 27989 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:30.800050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:48593 (local address 127.18.96.190:34945)
0928 14:48:30.800126 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:30.800129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:30.800215 (+ 86us) server_negotiation.cc:407] Connection header received
0928 14:48:30.800286 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:30.800287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:30.800302 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:30.800376 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:30.800723 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.801245 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.801985 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:30.802183 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:30.802270 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:30.804855 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:48593: BlockingRecv error: recv got EOF from 127.18.96.131:48593 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 14:48:31 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:31 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:31.540792 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:31.535907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:55873)
0928 14:48:31.536050 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:31.536066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:31.536144 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:31.536342 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:31.536343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:31.536538 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:31.536590 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:31.536591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.537241 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.537242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:31.537787 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:31.537788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.538112 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.538112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:31.538156 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:31.538606 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:31.538610 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:31.540733 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
W20260928 14:48:31.540917 27993 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:31.536041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:55873 (local address 127.18.96.190:34945)
0928 14:48:31.536199 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:31.536201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:31.536211 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:31.536237 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:31.536238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:31.536250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:31.536326 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:31.536664 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.537160 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.537858 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.538042 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.538090 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:31.540830 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:55873: BlockingRecv error: recv got EOF from 127.18.96.129:55873 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
Sep 28 14:48:31 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:31 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:31.672596 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:31.667726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:36125)
0928 14:48:31.667944 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:31.667959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:31.668043 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:31.668225 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:31.668226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:31.668423 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:31.668470 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:31.668472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.669158 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.669159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:31.669737 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:31.669739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.670120 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.670121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:31.670178 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:31.670625 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:31.670629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:31.672514 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":109,"threads_started":1}
W20260928 14:48:31.672775 27993 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:31.667818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:36125 (local address 127.18.96.190:34945)
0928 14:48:31.667992 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:31.667994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:31.668023 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:48:31.668128 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:31.668132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:31.668144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:31.668217 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:31.668559 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.669063 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.669831 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.670032 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.670119 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:31.672681 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:36125: BlockingRecv error: recv got EOF from 127.18.96.130:36125 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Sep 28 14:48:31 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:31 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.131@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:31.810724 27995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:31.805482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.131:57425)
0928 14:48:31.805710 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:31.805722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:31.805802 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:31.805971 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:31.805971 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:31.806164 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:31.806214 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:31.806216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.807124 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.807125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:31.807730 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:31.807732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.808273 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.808274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:31.808338 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:31.808789 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:31.808794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:31.810624 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":133,"threads_started":1}
W20260928 14:48:31.810776 27993 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:31.805599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:57425 (local address 127.18.96.190:34945)
0928 14:48:31.805694 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:31.805697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:31.805780 (+ 83us) server_negotiation.cc:407] Connection header received
0928 14:48:31.805873 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:31.805874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:31.805899 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:31.805962 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:31.806308 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.807056 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.807828 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:31.808149 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:31.808269 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:31.810708 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.131:57425: BlockingRecv error: recv got EOF from 127.18.96.131:57425 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 14:48:32 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:32 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.129@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:48:32.550875 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:32.541339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.129:55389)
0928 14:48:32.541557 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:32.541569 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:32.541626 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:32.541796 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:32.541797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:32.542000 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:32.542052 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:32.542086 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.542834 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.542835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:32.543455 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:32.543457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.543843 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.543844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:32.543975 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:32.548459 (+ 4484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:32.548465 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:32.550770 (+ 2305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":125,"threads_started":1}
W20260928 14:48:32.550979 27997 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:32.541477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:55389 (local address 127.18.96.190:34945)
0928 14:48:32.541624 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:32.541627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:32.541645 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:48:32.541672 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:32.541673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:32.541683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:32.541746 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:32.542167 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.542753 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.543546 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.543768 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.543828 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:32.550897 (+ 7069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.129:55389: BlockingRecv error: recv got EOF from 127.18.96.129:55389 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
Sep 28 14:48:32 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:48:32 dist-test-slave-j968 krb5kdc[26937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.130@KRBTEST.COM for oryx/127.18.96.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:48:32.678061 27997 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:32.673229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:40733 (local address 127.18.96.190:34945)
0928 14:48:32.673295 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:32.673297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:32.673376 (+ 79us) server_negotiation.cc:407] Connection header received
0928 14:48:32.673439 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:32.673440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:32.673450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:32.673500 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:32.673822 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.674327 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.675037 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.675261 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.675329 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:32.678009 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.130:40733: BlockingRecv error: recv got EOF from 127.18.96.130:40733 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:48:32.678059 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:32.673182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34945 (local address 127.18.96.130:40733)
0928 14:48:32.673296 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:32.673314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:32.673369 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:32.673546 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:32.673547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:32.673708 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:32.673756 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:32.673757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.674398 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.674398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:32.674935 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:32.674936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.675341 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.675342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:32.675387 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:32.675854 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:32.675858 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:32.677971 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.190:34945: Server oryx/127.18.96.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260928 14:48:32.727715 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 27520
I20260928 14:48:32.732481 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 27674
I20260928 14:48:32.736698 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 27806
I20260928 14:48:32.741161 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 27450
2026-09-28T14:48:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17474 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T14:48:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:32Z Disabled control of system clock
I20260928 14:48:32.769433 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:36423
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:48:32.850648 28002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:32.850849 28002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:32.850869 28002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:32.852072 28002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:32.852114 28002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:32.852128 28002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:32.852139 28002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:32.852151 28002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:32.853436 28002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:36423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:32.853637 28002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:32.853876 28002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:32.859548 28007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:32.859557 28008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:32.859572 28010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:32.859714 28002 server_base.cc:1034] running on GCE node
I20260928 14:48:32.860154 28002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:32.860468 28002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:32.861624 28002 hybrid_clock.cc:648] HybridClock initialized: now 1790606912861609 us; error 29 us; skew 500 ppm
I20260928 14:48:32.863016 28002 webserver.cc:460] Webserver started at http://127.18.96.190:36331/ using document root <none> and password file <none>
I20260928 14:48:32.863312 28002 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:32.863363 28002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:32.863562 28002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:32.864888 28002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "f9552d0af6f14a31a4a528a33932087b"
format_stamp: "Formatted at 2026-09-28 14:48:32 on dist-test-slave-j968"
server_key: "8f7641d3530648e45c4493ea57287404"
server_key_iv: "53e71c1975196235fd7b3711826f2708"
server_key_version: "encryptionkey@0"
I20260928 14:48:32.865288 28002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "f9552d0af6f14a31a4a528a33932087b"
format_stamp: "Formatted at 2026-09-28 14:48:32 on dist-test-slave-j968"
server_key: "8f7641d3530648e45c4493ea57287404"
server_key_iv: "53e71c1975196235fd7b3711826f2708"
server_key_version: "encryptionkey@0"
I20260928 14:48:32.866685 28002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:32.867507 28016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:32.867729 28002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:48:32.867811 28002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "f9552d0af6f14a31a4a528a33932087b"
format_stamp: "Formatted at 2026-09-28 14:48:32 on dist-test-slave-j968"
server_key: "8f7641d3530648e45c4493ea57287404"
server_key_iv: "53e71c1975196235fd7b3711826f2708"
server_key_version: "encryptionkey@0"
I20260928 14:48:32.867918 28002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:32.883476 28002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:32.883774 28002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:32.883968 28002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:32.887719 28002 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:36423
I20260928 14:48:32.887816 28068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:36423 every 8 connection(s)
I20260928 14:48:32.888208 28002 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:32.888938 28069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:32.891513 28069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b: Bootstrap starting.
I20260928 14:48:32.892185 28069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:32.892437 28069 log.cc:824] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:32.893180 28069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b: No bootstrap required, opened a new log
I20260928 14:48:32.893967 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 28002
I20260928 14:48:32.894048 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:32.894258 18818 external_mini_cluster.cc:1428] Setting key a55c6bf9792c62ce766eb9c07d025e2e
I20260928 14:48:32.895218 28069 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9552d0af6f14a31a4a528a33932087b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36423 } }
I20260928 14:48:32.895391 28069 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:32.895434 28069 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9552d0af6f14a31a4a528a33932087b, State: Initialized, Role: FOLLOWER
I20260928 14:48:32.895577 28069 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9552d0af6f14a31a4a528a33932087b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36423 } }
I20260928 14:48:32.895694 28069 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:32.895761 28069 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:32.895814 28069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:32.896582 28069 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9552d0af6f14a31a4a528a33932087b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36423 } }
I20260928 14:48:32.896701 28069 leader_election.cc:302] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9552d0af6f14a31a4a528a33932087b; no voters:
I20260928 14:48:32.896857 28069 leader_election.cc:288] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:32.896960 28074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:32.897092 28069 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:32.897095 28074 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [term 1 LEADER]: Becoming Leader. State: Replica: f9552d0af6f14a31a4a528a33932087b, State: Running, Role: LEADER
I20260928 14:48:32.897347 28074 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9552d0af6f14a31a4a528a33932087b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36423 } }
I20260928 14:48:32.897730 28075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9552d0af6f14a31a4a528a33932087b. Latest consensus state: current_term: 1 leader_uuid: "f9552d0af6f14a31a4a528a33932087b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9552d0af6f14a31a4a528a33932087b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36423 } } }
I20260928 14:48:32.897886 28075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:32.897746 28076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9552d0af6f14a31a4a528a33932087b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9552d0af6f14a31a4a528a33932087b" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36423 } } }
I20260928 14:48:32.898048 28076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:32.898299 28083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:32.898857 28083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:32.900729 28091 mvcc.cc:205] Tried to move back new op lower bound from 7334325915239620608 to 7334325915228004352. Current Snapshot: MvccSnapshot[applied={T|T < 7334325915239620608}]
I20260928 14:48:32.900816 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:32.894818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.18.96.190:36423)
0928 14:48:32.895087 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:32.895109 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:32.895142 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:48:32.895257 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:32.895264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:32.895333 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:32.895475 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:32.895733 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.897535 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.898585 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:32.899216 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:32.899357 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:32.900089 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:32.900092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:32.900093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:32.900265 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:32.900299 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:32.900310 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:32.900390 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:32.900469 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:48:32.900554 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260928 14:48:32.901204 28083 catalog_manager.cc:1347] Generated new cluster ID: 9c468dd4fee3410498502a86d70ca1d5
I20260928 14:48:32.901257 28083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:32.907261 28083 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:32.907778 28083 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:32.910359 28083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9552d0af6f14a31a4a528a33932087b: Generated new TSK 0
I20260928 14:48:32.910532 28083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:48:32.914813 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36423
--builtin_ntp_servers=127.18.96.148:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 14:48:33.010833 28093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:33.011055 28093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:33.011080 28093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:33.012374 28093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:33.012416 28093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:33.012445 28093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:33.013844 28093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:33.014106 28093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:33.014415 28093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:33.020560 28098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.020625 28099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.020613 28101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:33.020804 28093 server_base.cc:1034] running on GCE node
I20260928 14:48:33.021106 28093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:33.021358 28093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:33.022517 28093 hybrid_clock.cc:648] HybridClock initialized: now 1790606913022490 us; error 44 us; skew 500 ppm
I20260928 14:48:33.023839 28093 webserver.cc:460] Webserver started at http://127.18.96.129:44901/ using document root <none> and password file <none>
I20260928 14:48:33.024151 28093 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:33.024230 28093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:33.024380 28093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:33.025470 28093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "7e86e40bb1db4d728d3b41599a0cff61"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "7678191c33e6dce007dd68512192ad87"
server_key_iv: "8c42b0b59e0ece5906a296571046d00c"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.025832 28093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "7e86e40bb1db4d728d3b41599a0cff61"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "7678191c33e6dce007dd68512192ad87"
server_key_iv: "8c42b0b59e0ece5906a296571046d00c"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.027241 28093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:33.028080 28107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.028259 28093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:33.028354 28093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "7e86e40bb1db4d728d3b41599a0cff61"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "7678191c33e6dce007dd68512192ad87"
server_key_iv: "8c42b0b59e0ece5906a296571046d00c"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.028434 28093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:33.048623 28093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:33.048980 28093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:33.049151 28093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:33.049408 28093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:33.049767 28093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:33.049825 28093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.049867 28093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:33.049896 28093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.055756 28093 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:45875
I20260928 14:48:33.055835 28220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:45875 every 8 connection(s)
I20260928 14:48:33.056226 28093 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:33.060690 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 28093
I20260928 14:48:33.060770 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:33.061007 18818 external_mini_cluster.cc:1428] Setting key 5c52333619ccf6ca2df7427b0bb887ad
I20260928 14:48:33.061987 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36423
--builtin_ntp_servers=127.18.96.148:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:48:33.062234 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.056985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:60145 (local address 127.18.96.190:36423)
0928 14:48:33.057050 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:33.057053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:33.057412 (+ 359us) server_negotiation.cc:407] Connection header received
0928 14:48:33.057910 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:33.057911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:33.057927 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:33.057989 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:33.058687 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.059221 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.060284 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.060490 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.060575 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:33.061948 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:33.061949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:33.061949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:33.061956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:33.061975 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:33.061979 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:33.062044 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:33.062124 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 14:48:33.062155 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:48:33.062453 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.056989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:36423 (local address 127.18.96.129:60145)
0928 14:48:33.057211 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:33.057247 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:33.057797 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:33.058066 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:33.058069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:33.058078 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:33.058577 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:33.058581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.059311 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.059312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:33.060201 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:33.060203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.060913 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.060915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:33.061087 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:33.061780 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:33.061781 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:33.061821 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:33.061844 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:33.062044 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:33.062050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:33.062053 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:33.062127 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:48:33.062230 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
I20260928 14:48:33.062891 28221 heartbeater.cc:347] Connected to a master server at 127.18.96.190:36423
I20260928 14:48:33.062999 28221 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:33.063199 28221 heartbeater.cc:511] Master 127.18.96.190:36423 requested a full tablet report, sending...
I20260928 14:48:33.063757 28033 ts_manager.cc:194] Registered new tserver with Master: 7e86e40bb1db4d728d3b41599a0cff61 (127.18.96.129:45875)
I20260928 14:48:33.064420 28033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.129:60145
W20260928 14:48:33.144253 28224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:33.144474 28224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:33.144495 28224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:33.145732 28224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:33.145777 28224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:33.145803 28224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:33.147439 28224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:33.147717 28224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:33.148054 28224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:33.153620 28230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:33.153861 28224 server_base.cc:1034] running on GCE node
W20260928 14:48:33.153620 28232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.153682 28229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:33.154265 28224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:33.154527 28224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:33.155718 28224 hybrid_clock.cc:648] HybridClock initialized: now 1790606913155763 us; error 95 us; skew 500 ppm
I20260928 14:48:33.157217 28224 webserver.cc:460] Webserver started at http://127.18.96.130:33339/ using document root <none> and password file <none>
I20260928 14:48:33.157480 28224 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:33.157550 28224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:33.157707 28224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:33.159020 28224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "783c5ba562794e45adba78deb49a6b43"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "5a82d0e771ec7f990113356705ae79c0"
server_key_iv: "ea501ecae4d515c6ddbda845714264af"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.159431 28224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "783c5ba562794e45adba78deb49a6b43"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "5a82d0e771ec7f990113356705ae79c0"
server_key_iv: "ea501ecae4d515c6ddbda845714264af"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.160938 28224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:33.161826 28238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.162112 28224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:33.162197 28224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "783c5ba562794e45adba78deb49a6b43"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "5a82d0e771ec7f990113356705ae79c0"
server_key_iv: "ea501ecae4d515c6ddbda845714264af"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.162290 28224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:33.189257 28224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:33.189601 28224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:33.189749 28224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:33.189996 28224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:33.190337 28224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:33.190371 28224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.190395 28224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:33.190442 28224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.196200 28224 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:42945
I20260928 14:48:33.196274 28351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:42945 every 8 connection(s)
I20260928 14:48:33.196617 28224 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:33.197796 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 28224
I20260928 14:48:33.197896 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:33.198091 18818 external_mini_cluster.cc:1428] Setting key 70a8facd5bc655b32b391f4d2f8453ea
I20260928 14:48:33.199062 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36423
--builtin_ntp_servers=127.18.96.148:32771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:48:33.202684 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.197403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:35743 (local address 127.18.96.190:36423)
0928 14:48:33.197469 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:33.197472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:33.197763 (+ 291us) server_negotiation.cc:407] Connection header received
0928 14:48:33.198553 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:33.198554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:33.198566 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:33.198613 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:33.199451 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.200020 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.201017 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.201342 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.201415 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:33.202425 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:33.202426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:33.202426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:33.202432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:33.202446 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:33.202451 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:33.202498 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:33.202570 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 14:48:33.202603 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:48:33.202924 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.197403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:36423 (local address 127.18.96.130:35743)
0928 14:48:33.197643 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:33.197678 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:33.198405 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:33.198709 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:33.198713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:33.198724 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:33.199361 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:33.199367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.200111 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.200112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:33.200922 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:33.200924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.201482 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.201484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:33.201611 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:33.202279 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:33.202280 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:33.202309 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:33.202333 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:33.202497 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:33.202503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:33.202507 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:48:33.202574 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:48:33.202694 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260928 14:48:33.203424 28352 heartbeater.cc:347] Connected to a master server at 127.18.96.190:36423
I20260928 14:48:33.203536 28352 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:33.203732 28352 heartbeater.cc:511] Master 127.18.96.190:36423 requested a full tablet report, sending...
I20260928 14:48:33.204170 28033 ts_manager.cc:194] Registered new tserver with Master: 783c5ba562794e45adba78deb49a6b43 (127.18.96.130:42945)
I20260928 14:48:33.204734 28033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.130:35743
W20260928 14:48:33.283058 28355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:33.283289 28355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:33.283311 28355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:33.284454 28355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:33.284543 28355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:33.284588 28355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:33.285938 28355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:33.286139 28355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:33.286393 28355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:33.291550 28361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.291541 28360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.291541 28363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:33.291821 28355 server_base.cc:1034] running on GCE node
I20260928 14:48:33.292078 28355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:33.292352 28355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:33.293511 28355 hybrid_clock.cc:648] HybridClock initialized: now 1790606913293492 us; error 38 us; skew 500 ppm
I20260928 14:48:33.294807 28355 webserver.cc:460] Webserver started at http://127.18.96.131:36167/ using document root <none> and password file <none>
I20260928 14:48:33.295050 28355 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:33.295114 28355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:33.295286 28355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:33.296464 28355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "ae768f0bf7594ada870d3cae3b08549e"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "68a4399f95dccec384b1f131473e3671"
server_key_iv: "e6e9f25f3c06fb3f9844954a71e1bc30"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.296900 28355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "ae768f0bf7594ada870d3cae3b08549e"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "68a4399f95dccec384b1f131473e3671"
server_key_iv: "e6e9f25f3c06fb3f9844954a71e1bc30"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.298360 28355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:48:33.299165 28369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.299353 28355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:33.299453 28355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "ae768f0bf7594ada870d3cae3b08549e"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "68a4399f95dccec384b1f131473e3671"
server_key_iv: "e6e9f25f3c06fb3f9844954a71e1bc30"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.299522 28355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:33.318145 28355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:33.318502 28355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:33.318667 28355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:33.318907 28355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:33.319283 28355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:33.319339 28355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.319373 28355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:33.319399 28355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.325356 28355 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:44927
I20260928 14:48:33.325471 28482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:44927 every 8 connection(s)
I20260928 14:48:33.325794 28355 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:33.331272 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.326537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:34085 (local address 127.18.96.190:36423)
0928 14:48:33.326607 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:33.326611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:33.326876 (+ 265us) server_negotiation.cc:407] Connection header received
0928 14:48:33.327368 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:33.327369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:33.327379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:33.327428 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:33.328098 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.328609 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.329836 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.330021 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.330078 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:33.331052 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:33.331053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:33.331053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:33.331057 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:33.331070 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:33.331074 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:33.331109 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:33.331188 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:48:33.331212 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:48:33.331492 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.326528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:36423 (local address 127.18.96.131:34085)
0928 14:48:33.326748 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:33.326784 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:33.327281 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:33.327494 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:33.327497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:33.327505 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:33.328025 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:33.328029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.328707 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.328708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:33.329747 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:33.329750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:33.330111 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:33.330112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:33.330236 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:33.330854 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:33.330854 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:33.330883 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:33.330964 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:33.331121 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:33.331125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:33.331127 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:48:33.331182 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:48:33.331262 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
I20260928 14:48:33.332140 28483 heartbeater.cc:347] Connected to a master server at 127.18.96.190:36423
I20260928 14:48:33.332242 28483 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:33.332422 28483 heartbeater.cc:511] Master 127.18.96.190:36423 requested a full tablet report, sending...
I20260928 14:48:33.332772 28033 ts_manager.cc:194] Registered new tserver with Master: ae768f0bf7594ada870d3cae3b08549e (127.18.96.131:44927)
I20260928 14:48:33.333141 28033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.131:34085
I20260928 14:48:33.335417 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 28355
I20260928 14:48:33.335533 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:33.335743 18818 external_mini_cluster.cc:1428] Setting key 428e13b5bff6e4e9ae9bdb1b6d141c5b
I20260928 14:48:33.336588 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:48:33.338434 28073 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:33.337860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.18.96.190:36423)
0928 14:48:33.338003 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:33.338005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:33.338052 (+ 47us) server_negotiation.cc:407] Connection header received
0928 14:48:33.338140 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:33.338141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:33.338154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:33.338205 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:33.338342 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49696: BlockingRecv error: recv got EOF from 127.0.0.1:49696 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:48:33.338534 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 28093
I20260928 14:48:33.343648 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 28224
I20260928 14:48:33.347647 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 28355
I20260928 14:48:33.351917 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 28002
2026-09-28T14:48:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (609 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T14:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:33Z Disabled control of system clock
I20260928 14:48:33.378444 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:40861
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:40861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:48:33.460609 28494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:33.460810 28494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:33.460830 28494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:33.461913 28494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:33.461951 28494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:33.461963 28494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:33.461999 28494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:33.462023 28494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:33.463370 28494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:40861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:33.463554 28494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:33.463798 28494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:48:33.468680 28494 server_base.cc:1034] running on GCE node
W20260928 14:48:33.468649 28502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.468641 28500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.468649 28499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:33.469118 28494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:33.469357 28494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:33.470487 28494 hybrid_clock.cc:648] HybridClock initialized: now 1790606913470478 us; error 30 us; skew 500 ppm
I20260928 14:48:33.471692 28494 webserver.cc:460] Webserver started at http://127.18.96.190:36297/ using document root <none> and password file <none>
I20260928 14:48:33.471958 28494 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:33.472007 28494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:33.472102 28494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:33.473156 28494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "97f9bacdc60e4da58975e5cc2f0c76cb"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "62b0f954893291b69b28ba152134369c"
server_key_iv: "e427e2295c119e847f87dc4d3bbd3b0d"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.473448 28494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "97f9bacdc60e4da58975e5cc2f0c76cb"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "62b0f954893291b69b28ba152134369c"
server_key_iv: "e427e2295c119e847f87dc4d3bbd3b0d"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.474637 28494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:33.475274 28508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.475428 28494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:33.475498 28494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "97f9bacdc60e4da58975e5cc2f0c76cb"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "62b0f954893291b69b28ba152134369c"
server_key_iv: "e427e2295c119e847f87dc4d3bbd3b0d"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.475551 28494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:33.478822 28494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:33.479069 28494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:33.479167 28494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:33.483016 28494 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:40861
I20260928 14:48:33.483049 28560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:40861 every 8 connection(s)
I20260928 14:48:33.483381 28494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:33.484063 28561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:33.486788 28561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb: Bootstrap starting.
I20260928 14:48:33.487336 28561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:33.487634 28561 log.cc:824] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:33.488304 28561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb: No bootstrap required, opened a new log
I20260928 14:48:33.489864 28561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f9bacdc60e4da58975e5cc2f0c76cb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40861 } }
I20260928 14:48:33.490018 28561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:33.490077 28561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f9bacdc60e4da58975e5cc2f0c76cb, State: Initialized, Role: FOLLOWER
I20260928 14:48:33.490232 28561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f9bacdc60e4da58975e5cc2f0c76cb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40861 } }
I20260928 14:48:33.490307 28561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:33.490350 28561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:33.490396 28561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:33.490908 28561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f9bacdc60e4da58975e5cc2f0c76cb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40861 } }
I20260928 14:48:33.491017 28561 leader_election.cc:302] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97f9bacdc60e4da58975e5cc2f0c76cb; no voters:
I20260928 14:48:33.491178 28561 leader_election.cc:288] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:33.491212 28564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:33.491380 28564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [term 1 LEADER]: Becoming Leader. State: Replica: 97f9bacdc60e4da58975e5cc2f0c76cb, State: Running, Role: LEADER
I20260928 14:48:33.491444 28561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:33.491513 28564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f9bacdc60e4da58975e5cc2f0c76cb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40861 } }
I20260928 14:48:33.491881 28565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97f9bacdc60e4da58975e5cc2f0c76cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f9bacdc60e4da58975e5cc2f0c76cb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40861 } } }
I20260928 14:48:33.492053 28565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:33.492100 28566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97f9bacdc60e4da58975e5cc2f0c76cb. Latest consensus state: current_term: 1 leader_uuid: "97f9bacdc60e4da58975e5cc2f0c76cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f9bacdc60e4da58975e5cc2f0c76cb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40861 } } }
I20260928 14:48:33.492168 28566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:33.492533 28573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:33.492573 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 28494
I20260928 14:48:33.492668 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:33.492864 18818 external_mini_cluster.cc:1428] Setting key 489ad37ea318bb9cb102903f0b1e1cb6
I20260928 14:48:33.493132 28573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:33.494771 28573 catalog_manager.cc:1347] Generated new cluster ID: 9aa97508fc8b419d8414372eac965a70
I20260928 14:48:33.494814 28573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:33.497659 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.493329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.18.96.190:40861)
0928 14:48:33.496063 (+ 2734us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:33.496072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:33.496111 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:48:33.496221 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:33.496228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:33.496270 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:33.496403 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:33.497210 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:33.497214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:33.497214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:33.497218 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:33.497275 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:33.497288 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:33.497347 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:33.497435 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 14:48:33.497488 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2650,"thread_start_us":83,"threads_started":1}
I20260928 14:48:33.532030 28573 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:33.532254 28573 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:33.541472 28573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97f9bacdc60e4da58975e5cc2f0c76cb: Generated new TSK 0
I20260928 14:48:33.541690 28573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:48:33.551486 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40861
--builtin_ntp_servers=127.18.96.148:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:48:33.634346 28585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:33.634557 28585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:33.634578 28585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:33.635744 28585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:33.635784 28585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:33.635809 28585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:33.637251 28585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:33.637499 28585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:33.637835 28585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:48:33.643030 28585 server_base.cc:1034] running on GCE node
W20260928 14:48:33.642927 28591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.642927 28593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.642959 28590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:33.643483 28585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:33.643748 28585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:33.644918 28585 hybrid_clock.cc:648] HybridClock initialized: now 1790606913644902 us; error 45 us; skew 500 ppm
I20260928 14:48:33.646253 28585 webserver.cc:460] Webserver started at http://127.18.96.129:42985/ using document root <none> and password file <none>
I20260928 14:48:33.646507 28585 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:33.646570 28585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:33.646714 28585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:33.647780 28585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "55aaa7cd74384742ab17ca7956a8f004"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "31adfa15ec9aff2666212ffba8c8c8f3"
server_key_iv: "7b2f7ecd88823031538cc679594af9db"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.648206 28585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "55aaa7cd74384742ab17ca7956a8f004"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "31adfa15ec9aff2666212ffba8c8c8f3"
server_key_iv: "7b2f7ecd88823031538cc679594af9db"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.649480 28585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:33.650179 28599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.650379 28585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:33.650475 28585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "55aaa7cd74384742ab17ca7956a8f004"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "31adfa15ec9aff2666212ffba8c8c8f3"
server_key_iv: "7b2f7ecd88823031538cc679594af9db"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.650557 28585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:33.653888 28585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:33.654145 28585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:33.654276 28585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:33.654482 28585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:33.654801 28585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:33.654850 28585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.654888 28585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:33.654917 28585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.660565 28585 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:42391
I20260928 14:48:33.660609 28712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:42391 every 8 connection(s)
I20260928 14:48:33.660912 28585 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:33.664011 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.661619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:54767 (local address 127.18.96.190:40861)
0928 14:48:33.661677 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:33.661678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:33.661906 (+ 228us) server_negotiation.cc:407] Connection header received
0928 14:48:33.662418 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:33.662420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:33.662427 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:33.662486 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:33.663698 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:33.663700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:33.663700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:33.663705 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:33.663723 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:33.663727 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:33.663803 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:33.663922 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 14:48:33.663941 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:48:33.664201 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.661604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:40861 (local address 127.18.96.129:54767)
0928 14:48:33.661786 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:33.661817 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:33.662317 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:33.662558 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:33.662562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:33.662571 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:33.663522 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:33.663522 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:33.663550 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:33.663577 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:33.663803 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:33.663810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:33.663813 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:33.663922 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:48:33.663985 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260928 14:48:33.664674 28713 heartbeater.cc:347] Connected to a master server at 127.18.96.190:40861
I20260928 14:48:33.664763 28713 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:33.664906 28713 heartbeater.cc:511] Master 127.18.96.190:40861 requested a full tablet report, sending...
I20260928 14:48:33.665408 28525 ts_manager.cc:194] Registered new tserver with Master: 55aaa7cd74384742ab17ca7956a8f004 (127.18.96.129:42391)
I20260928 14:48:33.666788 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 28585
I20260928 14:48:33.666860 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:33.667011 18818 external_mini_cluster.cc:1428] Setting key 1b87d03fc6b0d50c4c0b05d182e2e2d9
I20260928 14:48:33.667834 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40861
--builtin_ntp_servers=127.18.96.148:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:48:33.751365 28716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:33.751657 28716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:33.751699 28716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:33.752981 28716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:33.753039 28716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:33.753083 28716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:33.754539 28716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:33.754812 28716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:33.755126 28716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:48:33.760663 28716 server_base.cc:1034] running on GCE node
W20260928 14:48:33.760663 28721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.760646 28722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.760640 28724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:33.761124 28716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:33.761360 28716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:33.762528 28716 hybrid_clock.cc:648] HybridClock initialized: now 1790606913762510 us; error 32 us; skew 500 ppm
I20260928 14:48:33.764036 28716 webserver.cc:460] Webserver started at http://127.18.96.130:37483/ using document root <none> and password file <none>
I20260928 14:48:33.764312 28716 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:33.764370 28716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:33.764547 28716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:33.765738 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "0918f8cb1f92428480d687972ace76b4"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "0d75e73ce5629296f00df2e2cf425716"
server_key_iv: "af7fdba70fbcfa492c56972cee3fe50c"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.766134 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "0918f8cb1f92428480d687972ace76b4"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "0d75e73ce5629296f00df2e2cf425716"
server_key_iv: "af7fdba70fbcfa492c56972cee3fe50c"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.767613 28716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:48:33.768476 28730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.768677 28716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:48:33.768774 28716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "0918f8cb1f92428480d687972ace76b4"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "0d75e73ce5629296f00df2e2cf425716"
server_key_iv: "af7fdba70fbcfa492c56972cee3fe50c"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.768849 28716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:33.772792 28716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:33.773099 28716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:33.773241 28716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:33.773456 28716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:33.773773 28716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:33.773808 28716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.773862 28716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:33.773895 28716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.780454 28716 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:34607
I20260928 14:48:33.780535 28843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:34607 every 8 connection(s)
I20260928 14:48:33.780895 28716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:33.782356 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 28716
I20260928 14:48:33.782478 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:33.782723 18818 external_mini_cluster.cc:1428] Setting key 275fcd16cf48b8bcda27d8c8e5687d3c
I20260928 14:48:33.783771 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40861
--builtin_ntp_servers=127.18.96.148:42405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:48:33.784411 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.781660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:53877 (local address 127.18.96.190:40861)
0928 14:48:33.781708 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:33.781710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:33.781995 (+ 285us) server_negotiation.cc:407] Connection header received
0928 14:48:33.782578 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:33.782580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:33.782588 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:33.782685 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:33.784131 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:33.784133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:33.784133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:33.784139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:33.784157 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:33.784162 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:33.784238 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:33.784288 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 14:48:33.784304 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:48:33.784626 28846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.781652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:40861 (local address 127.18.96.130:53877)
0928 14:48:33.781841 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:33.781887 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:33.782432 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:33.782735 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:33.782738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:33.782747 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:33.783844 (+ 1097us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:33.783845 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:33.783934 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:33.784048 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:33.784235 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:33.784237 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:33.784240 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:33.784294 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:48:33.784346 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260928 14:48:33.785116 28844 heartbeater.cc:347] Connected to a master server at 127.18.96.190:40861
I20260928 14:48:33.785243 28844 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:33.785411 28844 heartbeater.cc:511] Master 127.18.96.190:40861 requested a full tablet report, sending...
I20260928 14:48:33.785779 28525 ts_manager.cc:194] Registered new tserver with Master: 0918f8cb1f92428480d687972ace76b4 (127.18.96.130:34607)
W20260928 14:48:33.867187 28847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:33.867386 28847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:33.867408 28847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:33.868633 28847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:33.868674 28847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:33.868707 28847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:33.870019 28847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:33.870247 28847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:33.870532 28847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:33.875844 28853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.875859 28855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:33.875857 28852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:33.876121 28847 server_base.cc:1034] running on GCE node
I20260928 14:48:33.876335 28847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:33.876585 28847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:33.877744 28847 hybrid_clock.cc:648] HybridClock initialized: now 1790606913877738 us; error 34 us; skew 500 ppm
I20260928 14:48:33.879021 28847 webserver.cc:460] Webserver started at http://127.18.96.131:43271/ using document root <none> and password file <none>
I20260928 14:48:33.879261 28847 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:33.879326 28847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:33.879479 28847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:33.880653 28847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "9923a064a16d4058bcef56b28a7f6758"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "215e244d3ec068cb16219a3ff45f6d53"
server_key_iv: "5ced54aa088bb1d33fa3bcfcef14f9ea"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.881009 28847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "9923a064a16d4058bcef56b28a7f6758"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "215e244d3ec068cb16219a3ff45f6d53"
server_key_iv: "5ced54aa088bb1d33fa3bcfcef14f9ea"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.882251 28847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:48:33.882915 28861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.883074 28847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:48:33.883167 28847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "9923a064a16d4058bcef56b28a7f6758"
format_stamp: "Formatted at 2026-09-28 14:48:33 on dist-test-slave-j968"
server_key: "215e244d3ec068cb16219a3ff45f6d53"
server_key_iv: "5ced54aa088bb1d33fa3bcfcef14f9ea"
server_key_version: "encryptionkey@0"
I20260928 14:48:33.883234 28847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:33.886961 28847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:33.887221 28847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:33.887363 28847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:33.887580 28847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:33.887929 28847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:33.887977 28847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.888010 28847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:33.888036 28847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:33.893996 28847 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:37541
I20260928 14:48:33.894032 28974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:37541 every 8 connection(s)
I20260928 14:48:33.894392 28847 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:33.897555 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.895159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:44581 (local address 127.18.96.190:40861)
0928 14:48:33.895215 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:33.895217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:33.895449 (+ 232us) server_negotiation.cc:407] Connection header received
0928 14:48:33.896013 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:33.896015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:33.896021 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:33.896076 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:33.897290 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:33.897292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:33.897293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:48:33.897298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:33.897316 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:48:33.897320 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:33.897378 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:33.897471 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 14:48:33.897486 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 14:48:33.897747 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:33.895154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:40861 (local address 127.18.96.131:44581)
0928 14:48:33.895336 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:33.895369 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:33.895859 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:33.896163 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:33.896172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:33.896179 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:33.897135 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:48:33.897135 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:33.897162 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:48:33.897189 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:33.897376 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:33.897382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:33.897385 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:33.897477 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:48:33.897538 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260928 14:48:33.898260 28975 heartbeater.cc:347] Connected to a master server at 127.18.96.190:40861
I20260928 14:48:33.898353 28975 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:33.898515 28975 heartbeater.cc:511] Master 127.18.96.190:40861 requested a full tablet report, sending...
I20260928 14:48:33.898862 28525 ts_manager.cc:194] Registered new tserver with Master: 9923a064a16d4058bcef56b28a7f6758 (127.18.96.131:37541)
I20260928 14:48:33.899127 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 28847
I20260928 14:48:33.899250 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:33.899451 18818 external_mini_cluster.cc:1428] Setting key 0b740e6714ea42e13c0bb015de754779
I20260928 14:48:33.900363 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:48:33.902094 28582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:33.901649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.18.96.190:40861)
0928 14:48:33.901705 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:33.901707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:33.901812 (+ 105us) server_negotiation.cc:407] Connection header received
0928 14:48:33.901855 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:33.901856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:33.901860 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:33.901906 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:33.902021 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54976: BlockingRecv error: recv got EOF from 127.0.0.1:54976 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:48:33.902125 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 28585
I20260928 14:48:33.907361 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 28716
I20260928 14:48:33.911670 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 28847
I20260928 14:48:33.915601 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 28494
2026-09-28T14:48:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (563 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:33 dist-test-slave-j968 krb5kdc[28985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:33 dist-test-slave-j968 krb5kdc[28985](info): set up 2 sockets
Sep 28 14:48:33 dist-test-slave-j968 krb5kdc[28985](info): commencing operation
krb5kdc: starting...
W20260928 14:48:34.981853 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[28985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:48:35.107542 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:38467
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:38467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:35.190694 28998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:35.190919 28998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:35.190943 28998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:35.192245 28998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:35.192289 28998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:35.192303 28998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:35.192317 28998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:35.192328 28998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:35.193841 28998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:38467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:35.194113 28998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:35.194347 28998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:35.199425 29003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:35.199433 29004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:35.199440 29006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:35.199714 28998 server_base.cc:1034] running on GCE node
I20260928 14:48:35.200013 28998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:35.200259 28998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:35.201400 28998 hybrid_clock.cc:648] HybridClock initialized: now 1790606915201385 us; error 27 us; skew 500 ppm
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[28985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:35.204051 28998 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:48:35.204481 28998 webserver.cc:460] Webserver started at http://127.18.96.190:42641/ using document root <none> and password file <none>
I20260928 14:48:35.204715 28998 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:35.204779 28998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:35.204924 28998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:35.206008 28998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "406568d0151f9c58d79411a1ec90217a"
server_key_iv: "25bde8b46531e67e8ad23ec16e4a8121"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.206369 28998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "406568d0151f9c58d79411a1ec90217a"
server_key_iv: "25bde8b46531e67e8ad23ec16e4a8121"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.207739 28998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:48:35.208604 29013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.208784 28998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:48:35.208884 28998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "406568d0151f9c58d79411a1ec90217a"
server_key_iv: "25bde8b46531e67e8ad23ec16e4a8121"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.208982 28998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:35.236128 28998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:35.238111 28998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:35.238286 28998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:35.242446 28998 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:38467
I20260928 14:48:35.242482 29065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:38467 every 8 connection(s)
I20260928 14:48:35.242880 28998 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:35.243969 29066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:35.246357 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c: Bootstrap starting.
I20260928 14:48:35.246942 29066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:35.247256 29066 log.cc:824] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:35.248090 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c: No bootstrap required, opened a new log
I20260928 14:48:35.249752 29066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38467 } }
I20260928 14:48:35.249907 29066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:35.249931 29066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa34fdcd4494d35bd7bccf8e0e7416c, State: Initialized, Role: FOLLOWER
I20260928 14:48:35.250087 29066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38467 } }
I20260928 14:48:35.250176 29066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:35.250200 29066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:35.250231 29066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:35.251133 29066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38467 } }
I20260928 14:48:35.251232 29066 leader_election.cc:302] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aa34fdcd4494d35bd7bccf8e0e7416c; no voters:
I20260928 14:48:35.251366 29066 leader_election.cc:288] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:35.251449 29069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:35.251634 29069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [term 1 LEADER]: Becoming Leader. State: Replica: 1aa34fdcd4494d35bd7bccf8e0e7416c, State: Running, Role: LEADER
I20260928 14:48:35.251647 29066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:35.251840 29069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38467 } }
I20260928 14:48:35.252369 29070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38467 } } }
I20260928 14:48:35.252390 29071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aa34fdcd4494d35bd7bccf8e0e7416c. Latest consensus state: current_term: 1 leader_uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa34fdcd4494d35bd7bccf8e0e7416c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38467 } } }
I20260928 14:48:35.252485 29070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:35.252489 29071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:35.252983 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 28998
I20260928 14:48:35.253095 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:48:35.253343 18818 external_mini_cluster.cc:1428] Setting key 6a4f42fa3f35b672fdbe3b8bc6ba0b50
I20260928 14:48:35.252727 29078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:35.254094 29078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:35.255662 29078 catalog_manager.cc:1347] Generated new cluster ID: 6bb7d64e65914ceb9c8dce9864d35591
I20260928 14:48:35.255713 29078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[28985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:35.264995 29078 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:35.265599 29078 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:35.265771 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.253852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.18.96.190:38467)
0928 14:48:35.254028 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.254039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.254072 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:48:35.254213 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.254221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.254279 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.254392 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:35.255184 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.256588 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.257675 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.258309 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.258413 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.261469 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:35.261474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:35.261474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:35.261488 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:35.263626 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.264071 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.264075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.264076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.264212 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.264426 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.264427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.264427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.264717 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:35.264877 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.265333 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.265487 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260928 14:48:35.269424 29078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c: Generated new TSK 0
I20260928 14:48:35.269601 29078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:48:35.298743 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:38467
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:35.381850 29092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:35.382121 29092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:35.382160 29092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:35.383335 29092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:35.383404 29092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:35.383447 29092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:48:35.384948 29092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:35.385232 29092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:35.385466 29092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:35.390512 29098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:35.390714 29092 server_base.cc:1034] running on GCE node
W20260928 14:48:35.390501 29100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:35.390502 29097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:35.391026 29092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:35.391295 29092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:35.392436 29092 hybrid_clock.cc:648] HybridClock initialized: now 1790606915392421 us; error 28 us; skew 500 ppm
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[28985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:35.395061 29092 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:48:35.395561 29092 webserver.cc:460] Webserver started at http://127.18.96.129:46391/ using document root <none> and password file <none>
I20260928 14:48:35.395797 29092 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:35.395843 29092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:35.395998 29092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:35.397024 29092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "e83a870fb850415fa44f0811b5a6ee98"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "89e63597a03eea0c1484d16b82b8cc84"
server_key_iv: "224123a5d818e9799049794b0b101ad7"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.397333 29092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "e83a870fb850415fa44f0811b5a6ee98"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "89e63597a03eea0c1484d16b82b8cc84"
server_key_iv: "224123a5d818e9799049794b0b101ad7"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.398578 29092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:48:35.399281 29107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.399437 29092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:48:35.399529 29092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "e83a870fb850415fa44f0811b5a6ee98"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "89e63597a03eea0c1484d16b82b8cc84"
server_key_iv: "224123a5d818e9799049794b0b101ad7"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.399586 29092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:35.417850 29092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:35.420076 29092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:35.420192 29092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:35.420377 29092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:35.420703 29092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:35.420737 29092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.420799 29092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:35.420826 29092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.426759 29092 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:36951
I20260928 14:48:35.426812 29220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:36951 every 8 connection(s)
I20260928 14:48:35.427126 29092 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[28985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:35.437192 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.427845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:33159 (local address 127.18.96.190:38467)
0928 14:48:35.427906 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.427908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.428163 (+ 255us) server_negotiation.cc:407] Connection header received
0928 14:48:35.428706 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.428707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.428725 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.428772 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:35.429725 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.430223 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.431289 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.431501 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.431581 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.434154 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:35.434156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:35.434156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:35.434164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:35.435725 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.436206 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.436208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.436208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.436263 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.436504 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.436505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.436505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.436626 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:35.436676 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.437002 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.437085 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:48:35.437711 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.427807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:38467 (local address 127.18.96.129:33159)
0928 14:48:35.428024 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:35.428077 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:35.428589 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:35.428847 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:35.428851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:35.429427 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:35.429615 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.429620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.430369 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.430370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.431185 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.431187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.431592 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.431593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.431725 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.432304 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:35.432315 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:35.434027 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:35.435814 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:35.435821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:35.435823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:35.436113 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:35.436329 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:35.436330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:35.436330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:35.436441 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:35.436707 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:35.436712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:35.436863 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 14:48:35.437319 (+ 456us) client_negotiation.cc:240] Negotiation successful
0928 14:48:35.437492 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":132,"threads_started":1}
I20260928 14:48:35.439203 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 29092
I20260928 14:48:35.439297 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:35.439517 18818 external_mini_cluster.cc:1428] Setting key a3cc1fbd8a14c0263eaefb41a892e6ae
I20260928 14:48:35.439873 29221 heartbeater.cc:347] Connected to a master server at 127.18.96.190:38467
I20260928 14:48:35.439990 29221 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:35.440179 29221 heartbeater.cc:511] Master 127.18.96.190:38467 requested a full tablet report, sending...
I20260928 14:48:35.440901 29030 ts_manager.cc:194] Registered new tserver with Master: e83a870fb850415fa44f0811b5a6ee98 (127.18.96.129:36951)
I20260928 14:48:35.441819 29030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:33159
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:48:35.467100 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:38467
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:35.550633 29226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:35.550851 29226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:35.550875 29226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:35.552135 29226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:35.552188 29226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:35.552217 29226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:48:35.553764 29226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:35.553967 29226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:35.554154 29226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:35.559437 29234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:35.559437 29232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:35.559437 29231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:35.559671 29226 server_base.cc:1034] running on GCE node
I20260928 14:48:35.559927 29226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:35.560209 29226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:35.561403 29226 hybrid_clock.cc:648] HybridClock initialized: now 1790606915561388 us; error 29 us; skew 500 ppm
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[28985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:35.564095 29226 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:48:35.564613 29226 webserver.cc:460] Webserver started at http://127.18.96.130:43801/ using document root <none> and password file <none>
I20260928 14:48:35.564842 29226 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:35.564906 29226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:35.565043 29226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:35.566161 29226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "050a178683964d62bdcc22d20c46bf38"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "5d57cc3e8ac79916bb898678e336029c"
server_key_iv: "ddf1ae5227cecee4b9708c3967fbc7ad"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.566533 29226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "050a178683964d62bdcc22d20c46bf38"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "5d57cc3e8ac79916bb898678e336029c"
server_key_iv: "ddf1ae5227cecee4b9708c3967fbc7ad"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.567924 29226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 14:48:35.568743 29241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.568908 29226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:48:35.569015 29226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "050a178683964d62bdcc22d20c46bf38"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "5d57cc3e8ac79916bb898678e336029c"
server_key_iv: "ddf1ae5227cecee4b9708c3967fbc7ad"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.569137 29226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:35.577171 29226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:35.579456 29226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:35.579619 29226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:35.579849 29226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:35.580204 29226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:35.580255 29226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.580289 29226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:35.580317 29226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.586277 29226 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:35643
I20260928 14:48:35.586323 29354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:35643 every 8 connection(s)
I20260928 14:48:35.586629 29226 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:35.592083 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 29226
I20260928 14:48:35.592195 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:35.592401 18818 external_mini_cluster.cc:1428] Setting key 777de614a0edb33c91a3ac52c91c28b6
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[28985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:35.596418 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.587313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:48223 (local address 127.18.96.190:38467)
0928 14:48:35.587393 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.587396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.587575 (+ 179us) server_negotiation.cc:407] Connection header received
0928 14:48:35.588119 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.588121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.588133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.588183 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:35.589121 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.589643 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.590640 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.590857 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.590942 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.593321 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:35.593323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:35.593323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:35.593330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:35.594892 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.595305 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.595307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.595307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.595380 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.595616 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.595617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.595618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.595766 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:35.595839 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.596269 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.596349 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:48:35.596735 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.587284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:38467 (local address 127.18.96.130:48223)
0928 14:48:35.587454 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:35.587490 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:35.588015 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:35.588259 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:35.588263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:35.588828 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:35.589020 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.589024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.589738 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.589739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.590543 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.590545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.590951 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.590952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.591076 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.591537 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:35.591544 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:35.593213 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:35.594980 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:35.594986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:35.594988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:35.595230 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:35.595447 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:35.595448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:35.595448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:35.595550 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:35.595940 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:35.595947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:35.596116 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 14:48:35.596237 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 14:48:35.596419 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260928 14:48:35.597332 29355 heartbeater.cc:347] Connected to a master server at 127.18.96.190:38467
I20260928 14:48:35.597450 29355 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:35.597647 29355 heartbeater.cc:511] Master 127.18.96.190:38467 requested a full tablet report, sending...
I20260928 14:48:35.598030 29030 ts_manager.cc:194] Registered new tserver with Master: 050a178683964d62bdcc22d20c46bf38 (127.18.96.130:35643)
I20260928 14:48:35.598652 29030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:48223
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:48:35.619096 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:38467
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:35.702088 29360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:35.702304 29360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:35.702328 29360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:35.703578 29360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:35.703624 29360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:35.703653 29360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:48:35.705120 29360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:48:35.705370 29360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:35.705621 29360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:35.711194 29366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:35.711194 29368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:35.711822 29365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:35.711776 29360 server_base.cc:1034] running on GCE node
I20260928 14:48:35.712093 29360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:35.712323 29360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:35.713521 29360 hybrid_clock.cc:648] HybridClock initialized: now 1790606915713481 us; error 43 us; skew 500 ppm
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[28985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:35.717105 29360 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:48:35.717586 29360 webserver.cc:460] Webserver started at http://127.18.96.131:44875/ using document root <none> and password file <none>
I20260928 14:48:35.717872 29360 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:35.717957 29360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:35.718101 29360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:35.719427 29360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "e4a3393872c048ef86bd37d79f219e8c"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "0a986404ee2bf3fe33fe47897d61ff53"
server_key_iv: "9a43231f468528799488f2f87f2b368d"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.719816 29360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "e4a3393872c048ef86bd37d79f219e8c"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "0a986404ee2bf3fe33fe47897d61ff53"
server_key_iv: "9a43231f468528799488f2f87f2b368d"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.721268 29360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:35.722112 29375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.722318 29360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:35.722422 29360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "e4a3393872c048ef86bd37d79f219e8c"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-j968"
server_key: "0a986404ee2bf3fe33fe47897d61ff53"
server_key_iv: "9a43231f468528799488f2f87f2b368d"
server_key_version: "encryptionkey@0"
I20260928 14:48:35.722510 29360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:35.745419 29360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:35.747988 29360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:35.748189 29360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:35.748459 29360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:35.748796 29360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:35.748848 29360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.748899 29360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:35.748926 29360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.755157 29360 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:44805
I20260928 14:48:35.755230 29488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:44805 every 8 connection(s)
I20260928 14:48:35.755568 29360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[28985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606915, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:48:35.764295 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 29360
I20260928 14:48:35.764427 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:35.764638 18818 external_mini_cluster.cc:1428] Setting key 20b24e2ec401d9d419d46da3574bd579
I20260928 14:48:35.765484 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.756361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:37357 (local address 127.18.96.190:38467)
0928 14:48:35.756430 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.756432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.756647 (+ 215us) server_negotiation.cc:407] Connection header received
0928 14:48:35.757212 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.757213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.757222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.757284 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:35.758246 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.758734 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.759835 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.760100 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.760182 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.762550 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:35.762552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:35.762552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:35.762560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:35.763805 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.764365 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.764367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.764367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.764441 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.764738 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.764739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.764740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.764897 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:35.764960 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.765293 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.765381 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:48:35.765756 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.756336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:38467 (local address 127.18.96.131:37357)
0928 14:48:35.756515 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:35.756555 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:35.757119 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:35.757341 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:35.757345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:35.757956 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:35.758169 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.758174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.758822 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.758823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.759756 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.759758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.760199 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.760200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.760317 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.760875 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:35.760884 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:35.762439 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:35.764043 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:35.764049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:35.764052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:35.764286 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:35.764530 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:35.764531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:35.764531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:35.764653 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:35.764957 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:35.764963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:35.765146 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 14:48:35.765270 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:48:35.765482 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260928 14:48:35.766268 29489 heartbeater.cc:347] Connected to a master server at 127.18.96.190:38467
I20260928 14:48:35.766387 29489 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:35.766574 29489 heartbeater.cc:511] Master 127.18.96.190:38467 requested a full tablet report, sending...
I20260928 14:48:35.766918 29030 ts_manager.cc:194] Registered new tserver with Master: e4a3393872c048ef86bd37d79f219e8c (127.18.96.131:44805)
I20260928 14:48:35.767290 29030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:37357
I20260928 14:48:35.776678 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:48:35.783798 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.778002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.18.96.190:38467)
0928 14:48:35.778068 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.778070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.778185 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:48:35.778226 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.778227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.778235 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.778285 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:35.778806 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.779280 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.780019 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.780201 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.780254 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.781415 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:35.781416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:35.781416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:35.781420 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:35.782703 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.783061 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.783061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.783062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.783111 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.783321 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.783322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.783322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.783457 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:35.783528 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.783644 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.783705 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:48:35.785495 29030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:35.787596 29030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:35.794297 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.790220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.18.96.130:35643)
0928 14:48:35.790500 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.790505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.790564 (+ 59us) server_negotiation.cc:407] Connection header received
0928 14:48:35.790696 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.790699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.790715 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.790793 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:35.791031 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.792196 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.793294 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.793911 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.793982 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.794038 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.794168 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.794213 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260928 14:48:35.794287 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.790131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:35643 (local address 127.0.0.1:51296)
0928 14:48:35.790452 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:35.790495 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:35.790595 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:35.790791 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:35.790792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:35.790794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:35.790856 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.790858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.792317 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.792318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.793188 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.793191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.793979 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.793979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.794054 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.794066 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 14:48:35.794149 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:48:35.794206 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":43,"thread_start_us":57,"threads_started":1}
I20260928 14:48:35.794955 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.790334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:44805 (local address 127.0.0.1:44510)
0928 14:48:35.790840 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:35.790850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:35.790902 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:35.791210 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:35.791210 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:35.791213 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:35.791266 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.791268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.792199 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.792200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.793557 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.793560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.794706 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.794708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.794778 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.794783 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:48:35.794844 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 14:48:35.794888 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":69,"thread_start_us":73,"threads_started":1}
I20260928 14:48:35.794993 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.790433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.18.96.131:44805)
0928 14:48:35.790718 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.790722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.790896 (+ 174us) server_negotiation.cc:407] Connection header received
0928 14:48:35.790995 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.790998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.791013 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.791082 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:35.791342 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.792106 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.793654 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.794630 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.794695 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.794745 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.794861 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.794902 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":74,"threads_started":1}
I20260928 14:48:35.795120 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.790160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36951 (local address 127.0.0.1:41230)
0928 14:48:35.790456 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:35.790498 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:35.790563 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:35.790748 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:35.790751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:35.790753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:35.790842 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.790845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.792282 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.792282 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.793595 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.793597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.794920 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.794921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.794980 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.794985 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:48:35.795014 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 14:48:35.795051 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":52,"thread_start_us":89,"threads_started":1}
I20260928 14:48:35.795265 29289 tablet_service.cc:1467] Processing CreateTablet for tablet dc3f26734fe24df2be43818c67452c03 (DEFAULT_TABLE table=test-table [id=6b3e90dcce1e4a199cd142fbf88541b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:35.795651 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3f26734fe24df2be43818c67452c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:35.795666 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.790258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.18.96.129:36951)
0928 14:48:35.790593 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.790598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.790608 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:35.790651 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.790653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.790670 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.790746 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:35.790941 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.792185 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.794074 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.794825 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.795445 (+ 620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.795510 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.795536 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.795582 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":89,"threads_started":1}
I20260928 14:48:35.795753 29423 tablet_service.cc:1467] Processing CreateTablet for tablet dc3f26734fe24df2be43818c67452c03 (DEFAULT_TABLE table=test-table [id=6b3e90dcce1e4a199cd142fbf88541b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:35.796253 29423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3f26734fe24df2be43818c67452c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:35.796398 29155 tablet_service.cc:1467] Processing CreateTablet for tablet dc3f26734fe24df2be43818c67452c03 (DEFAULT_TABLE table=test-table [id=6b3e90dcce1e4a199cd142fbf88541b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:35.796702 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3f26734fe24df2be43818c67452c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:35.798699 29506 tablet_bootstrap.cc:492] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: Bootstrap starting.
I20260928 14:48:35.799562 29507 tablet_bootstrap.cc:492] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: Bootstrap starting.
I20260928 14:48:35.799779 29506 tablet_bootstrap.cc:654] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:35.800068 29508 tablet_bootstrap.cc:492] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98: Bootstrap starting.
I20260928 14:48:35.800251 29507 tablet_bootstrap.cc:654] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:35.800403 29506 log.cc:824] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:35.800768 29508 tablet_bootstrap.cc:654] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:35.800830 29507 log.cc:824] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:35.801231 29506 tablet_bootstrap.cc:492] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: No bootstrap required, opened a new log
I20260928 14:48:35.801312 29506 ts_tablet_manager.cc:1397] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:48:35.801327 29508 log.cc:824] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:35.801678 29507 tablet_bootstrap.cc:492] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: No bootstrap required, opened a new log
I20260928 14:48:35.801788 29507 ts_tablet_manager.cc:1397] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:35.802222 29508 tablet_bootstrap.cc:492] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98: No bootstrap required, opened a new log
I20260928 14:48:35.802291 29508 ts_tablet_manager.cc:1397] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:48:35.803363 29506 raft_consensus.cc:348] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } }
I20260928 14:48:35.803520 29506 raft_consensus.cc:374] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:35.803562 29506 raft_consensus.cc:729] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a3393872c048ef86bd37d79f219e8c, State: Initialized, Role: FOLLOWER
I20260928 14:48:35.803694 29506 consensus_queue.cc:260] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } }
I20260928 14:48:35.803978 29506 ts_tablet_manager.cc:1428] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:35.803901 29507 raft_consensus.cc:348] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } }
I20260928 14:48:35.804081 29507 raft_consensus.cc:374] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:35.804107 29489 heartbeater.cc:503] Master 127.18.96.190:38467 was elected leader, sending a full tablet report...
I20260928 14:48:35.804123 29507 raft_consensus.cc:729] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050a178683964d62bdcc22d20c46bf38, State: Initialized, Role: FOLLOWER
I20260928 14:48:35.804245 29507 consensus_queue.cc:260] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } }
I20260928 14:48:35.804297 29508 raft_consensus.cc:348] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } }
I20260928 14:48:35.804457 29508 raft_consensus.cc:374] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:35.804504 29508 raft_consensus.cc:729] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83a870fb850415fa44f0811b5a6ee98, State: Initialized, Role: FOLLOWER
I20260928 14:48:35.804522 29507 ts_tablet_manager.cc:1428] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:35.804632 29355 heartbeater.cc:503] Master 127.18.96.190:38467 was elected leader, sending a full tablet report...
I20260928 14:48:35.804622 29508 consensus_queue.cc:260] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } }
I20260928 14:48:35.804874 29508 ts_tablet_manager.cc:1428] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:48:35.804991 29221 heartbeater.cc:503] Master 127.18.96.190:38467 was elected leader, sending a full tablet report...
W20260928 14:48:35.837168 29356 tablet.cc:2369] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:35.870276 29514 raft_consensus.cc:482] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:35.870411 29514 raft_consensus.cc:504] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } }
I20260928 14:48:35.870750 29514 leader_election.cc:288] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 050a178683964d62bdcc22d20c46bf38 (127.18.96.130:35643), e4a3393872c048ef86bd37d79f219e8c (127.18.96.131:44805)
I20260928 14:48:35.873855 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.870943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:35643 (local address 127.18.96.129:49925)
0928 14:48:35.871030 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:35.871045 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:35.871115 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:35.871276 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:35.871277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:35.871278 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:35.871372 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.871374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.872159 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.872160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.872991 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.872994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.873574 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.873574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.873659 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.873662 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:35.873737 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:48:35.873778 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:48:35.873855 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.871033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:49925 (local address 127.18.96.130:35643)
0928 14:48:35.871087 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.871089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.871099 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:48:35.871187 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.871188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.871199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.871258 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:35.871475 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.872063 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.873088 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.873488 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.873568 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.873577 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.873749 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.873787 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:48:35.873978 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.871159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:35773 (local address 127.18.96.131:44805)
0928 14:48:35.871215 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.871218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.871327 (+ 109us) server_negotiation.cc:407] Connection header received
0928 14:48:35.871401 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.871402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.871410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.871478 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:35.871634 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.872326 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.873248 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.873639 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.873725 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.873733 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.873875 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.873913 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:48:35.873991 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.871089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:44805 (local address 127.18.96.129:35773)
0928 14:48:35.871260 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:35.871268 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:35.871336 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:35.871495 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:35.871495 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:35.871497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:35.871568 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.871570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.872424 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.872425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.873155 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:35.873157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.873725 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.873726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:35.873804 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.873807 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:48:35.873870 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:48:35.873913 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":112,"threads_started":1}
I20260928 14:48:35.874224 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3f26734fe24df2be43818c67452c03" candidate_uuid: "e83a870fb850415fa44f0811b5a6ee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050a178683964d62bdcc22d20c46bf38" is_pre_election: true
I20260928 14:48:35.874413 29309 raft_consensus.cc:2454] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e83a870fb850415fa44f0811b5a6ee98 in term 0.
I20260928 14:48:35.874694 29108 leader_election.cc:302] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050a178683964d62bdcc22d20c46bf38, e83a870fb850415fa44f0811b5a6ee98; no voters:
I20260928 14:48:35.874842 29514 raft_consensus.cc:2781] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:35.874814 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3f26734fe24df2be43818c67452c03" candidate_uuid: "e83a870fb850415fa44f0811b5a6ee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4a3393872c048ef86bd37d79f219e8c" is_pre_election: true
I20260928 14:48:35.874894 29514 raft_consensus.cc:482] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:35.874931 29514 raft_consensus.cc:3037] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:35.874963 29443 raft_consensus.cc:2454] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e83a870fb850415fa44f0811b5a6ee98 in term 0.
I20260928 14:48:35.875676 29514 raft_consensus.cc:504] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } }
I20260928 14:48:35.875782 29514 leader_election.cc:288] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [CANDIDATE]: Term 1 election: Requested vote from peers 050a178683964d62bdcc22d20c46bf38 (127.18.96.130:35643), e4a3393872c048ef86bd37d79f219e8c (127.18.96.131:44805)
I20260928 14:48:35.876166 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3f26734fe24df2be43818c67452c03" candidate_uuid: "e83a870fb850415fa44f0811b5a6ee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050a178683964d62bdcc22d20c46bf38"
I20260928 14:48:35.876250 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3f26734fe24df2be43818c67452c03" candidate_uuid: "e83a870fb850415fa44f0811b5a6ee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4a3393872c048ef86bd37d79f219e8c"
I20260928 14:48:35.876255 29309 raft_consensus.cc:3037] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:35.876354 29443 raft_consensus.cc:3037] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:35.877187 29443 raft_consensus.cc:2454] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e83a870fb850415fa44f0811b5a6ee98 in term 1.
I20260928 14:48:35.877189 29309 raft_consensus.cc:2454] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e83a870fb850415fa44f0811b5a6ee98 in term 1.
I20260928 14:48:35.877341 29108 leader_election.cc:302] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050a178683964d62bdcc22d20c46bf38, e83a870fb850415fa44f0811b5a6ee98; no voters:
I20260928 14:48:35.877447 29514 raft_consensus.cc:2781] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:35.877595 29514 raft_consensus.cc:686] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 1 LEADER]: Becoming Leader. State: Replica: e83a870fb850415fa44f0811b5a6ee98, State: Running, Role: LEADER
I20260928 14:48:35.877696 29514 consensus_queue.cc:237] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } }
I20260928 14:48:35.878707 29029 catalog_manager.cc:5495] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 reported cstate change: term changed from 0 to 1, leader changed from <none> to e83a870fb850415fa44f0811b5a6ee98 (127.18.96.129). New cstate: current_term: 1 leader_uuid: "e83a870fb850415fa44f0811b5a6ee98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e83a870fb850415fa44f0811b5a6ee98" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36951 } health_report { overall_health: HEALTHY } } }
I20260928 14:48:35.887820 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.885343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.18.96.129:36951)
0928 14:48:35.885393 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.885395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.885407 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:48:35.885446 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.885447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.885467 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.885551 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:35.885715 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.886211 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.886985 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.887201 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.887272 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.887461 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:35.887572 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:35.887668 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.887687 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.887709 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:48:35.889659 29443 raft_consensus.cc:1261] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [term 1 FOLLOWER]: Refusing update from remote peer e83a870fb850415fa44f0811b5a6ee98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:35.889673 29309 raft_consensus.cc:1261] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [term 1 FOLLOWER]: Refusing update from remote peer e83a870fb850415fa44f0811b5a6ee98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:35.890064 29514 consensus_queue.cc:1035] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a3393872c048ef86bd37d79f219e8c" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:35.890208 29516 consensus_queue.cc:1035] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "050a178683964d62bdcc22d20c46bf38" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:35.892800 29519 mvcc.cc:205] Tried to move back new op lower bound from 7334325927481372672 to 7334325927435890688. Current Snapshot: MvccSnapshot[applied={T|T < 7334325927481372672}]
I20260928 14:48:35.892989 29521 mvcc.cc:205] Tried to move back new op lower bound from 7334325927481372672 to 7334325927435890688. Current Snapshot: MvccSnapshot[applied={T|T < 7334325927481372672}]
I20260928 14:48:35.895224 29030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:35.895314 29030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:35.896216 29030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1aa34fdcd4494d35bd7bccf8e0e7416c: Sending DeleteTablet for 3 replicas of tablet dc3f26734fe24df2be43818c67452c03
I20260928 14:48:35.896562 29289 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3f26734fe24df2be43818c67452c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:35 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:51296
I20260928 14:48:35.896576 29423 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3f26734fe24df2be43818c67452c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:35 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:44510
I20260928 14:48:35.896790 29529 tablet_replica.cc:330] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: stopping tablet replica
I20260928 14:48:35.896893 29529 raft_consensus.cc:2229] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:35.896893 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3f26734fe24df2be43818c67452c03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:35 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:41230
I20260928 14:48:35.897019 29529 raft_consensus.cc:2258] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:35.897094 29530 tablet_replica.cc:330] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: stopping tablet replica
I20260928 14:48:35.897185 29530 raft_consensus.cc:2229] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:35.897276 29530 raft_consensus.cc:2258] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:35.897439 29531 tablet_replica.cc:330] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98: stopping tablet replica
I20260928 14:48:35.897473 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 29092
I20260928 14:48:35.897486 29529 ts_tablet_manager.cc:1905] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:35.897524 29531 raft_consensus.cc:2229] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:48:35.897626 29531 raft_consensus.cc:2258] T dc3f26734fe24df2be43818c67452c03 P e83a870fb850415fa44f0811b5a6ee98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:35.899698 29529 ts_tablet_manager.cc:1918] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:35.899751 29529 log.cc:1196] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/dc3f26734fe24df2be43818c67452c03
I20260928 14:48:35.900104 29529 ts_tablet_manager.cc:1939] T dc3f26734fe24df2be43818c67452c03 P 050a178683964d62bdcc22d20c46bf38: Deleting consensus metadata
I20260928 14:48:35.900522 29014 catalog_manager.cc:4841] TS 050a178683964d62bdcc22d20c46bf38 (127.18.96.130:35643): tablet dc3f26734fe24df2be43818c67452c03 (table test-table [id=6b3e90dcce1e4a199cd142fbf88541b9]) successfully deleted
W20260928 14:48:35.903615 29017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:36951 (error 108)
I20260928 14:48:35.903831 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 29226
I20260928 14:48:35.904479 29530 ts_tablet_manager.cc:1905] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:35.906453 29530 ts_tablet_manager.cc:1918] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:35.906533 29530 log.cc:1196] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/dc3f26734fe24df2be43818c67452c03
I20260928 14:48:35.906809 29530 ts_tablet_manager.cc:1939] T dc3f26734fe24df2be43818c67452c03 P e4a3393872c048ef86bd37d79f219e8c: Deleting consensus metadata
I20260928 14:48:35.907418 29014 catalog_manager.cc:4841] TS e4a3393872c048ef86bd37d79f219e8c (127.18.96.131:44805): tablet dc3f26734fe24df2be43818c67452c03 (table test-table [id=6b3e90dcce1e4a199cd142fbf88541b9]) successfully deleted
I20260928 14:48:35.908099 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.904485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36951 (local address 127.0.0.1:41246)
0928 14:48:35.908023 (+ 3538us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:35.908069 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3516}
W20260928 14:48:35.908272 29017 catalog_manager.cc:4584] TS e83a870fb850415fa44f0811b5a6ee98 (127.18.96.129:36951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc3f26734fe24df2be43818c67452c03: Network error: Client connection negotiation failed: client connection to 127.18.96.129:36951: connect: Connection refused (error 111)
I20260928 14:48:35.909107 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 29360
I20260928 14:48:35.914218 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 28998
2026-09-28T14:48:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1999 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[29535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[29535](info): set up 2 sockets
Sep 28 14:48:35 dist-test-slave-j968 krb5kdc[29535](info): commencing operation
krb5kdc: starting...
W20260928 14:48:36.961815 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:48:37 dist-test-slave-j968 krb5kdc[29535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.96.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.96.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/rangeradmin_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/rangeradmin_127.18.96.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.96.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.96.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/rangerlookup_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/rangerlookup_127.18.96.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.96.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/HTTP_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/HTTP_127.18.96.148@KRBTEST.COM.keytab.
I20260928 14:48:37.131171 18818 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklohQF_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/postgres -l logfile start
2026-09-28 14:48:39.754 UTC [29565] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:48:39.754 UTC [29565] LOG: listening on IPv4 address "127.18.96.148", port 50685
2026-09-28 14:48:39.758 UTC [29565] LOG: listening on Unix socket "/tmp/.s.PGSQL.50685"
2026-09-28 14:48:39.795 UTC [29568] LOG: database system was shut down at 2026-09-28 14:48:38 UTC
2026-09-28 14:48:39.799 UTC [29565] LOG: database system is ready to accept connections
I20260928 14:48:40.753711 18818 mini_postgres.cc:96] Postgres bound to 50685
2026-09-28 14:48:40.760 UTC [29576] FATAL: database "slave" does not exist
127.18.96.148:50685 - accepting connections
I20260928 14:48:40.760754 18818 mini_ranger.cc:161] Starting Ranger...
I20260928 14:48:40.776502 18818 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:48:41.312613 18818 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:48:41.312703 18818 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-admin
2026-09-28 14:48:41,624 [I] DB FLAVOR :POSTGRES
2026-09-28 14:48:41,625 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:48:41,625 [I] Checking connection..
2026-09-28 14:48:41,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:48:41,896 [I] Checking connection passed.
2026-09-28 14:48:41,896 [I] --------- Verifying version history table ---------
2026-09-28 14:48:41,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:48:42,145 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:48:42,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:48:42,402 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:48:42,403 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:48:42,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:48:42.646 UTC [29640] WARNING: there is no transaction in progress
2026-09-28 14:48:42,652 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:48:42,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:48:42,890 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:48:42,890 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:48:42,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:48:43,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:48:43,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j968.c.gcp-upstream.internal','N') ;"
2026-09-28 14:48:43,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:48:43,883 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:48:43,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:48:44,165 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:48:44,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:48:44,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:48:44,656 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:48:44,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:48:45.631 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.648 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.664 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.680 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.697 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.719 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.792 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.800 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.816 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.828 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.839 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.853 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.863 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.869 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.877 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:45.884 UTC [29748] WARNING: there is no transaction in progress
2026-09-28 14:48:46,550 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:48:46,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:48:46,778 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:48:46,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:48:47,019 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:48:47,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:48:47,271 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:48:47,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:48:47,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:48:47,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j968.c.gcp-upstream.internal';"
2026-09-28 14:48:47,975 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:48:47,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:48:48,205 [I] DB_PATCHES have already been applied
I20260928 14:48:48.211314 18818 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklohQF_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:48:48.211382 18818 mini_ranger.cc:193] Using host: 127.18.96.148
I20260928 14:48:48.214778 18818 mini_ranger.cc:227] Ranger admin URL: http://127.18.96.148:43395
Sep 28, 2026 2:48:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:48:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:48:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:48:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:48:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:48:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:48:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:48:49.359107 18818 mini_ranger.cc:160] Time spent starting Ranger: real 8.598s user 0.004s sys 0.005s
Sep 28, 2026 2:49:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:49:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 2:49:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:49:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:49:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:49:11.180271 18818 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.96.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.96.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/rangerkms_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/rangerkms_127.18.96.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.96.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.96.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/HTTP_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.96.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/HTTP_127.18.96.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:49:11 dist-test-slave-j968 krb5kdc[29535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606951, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:49:11.270049 18818 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:49:11.279495 18818 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:49:12.355263 18818 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:49:12.355350 18818 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-kms
I20260928 14:49:12.355371 18818 mini_ranger_kms.cc:227] Using postgres at 127.18.96.148:50685
2026-09-28 14:49:12,888 [I] DB FLAVOR :POSTGRES
2026-09-28 14:49:12,889 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:49:12,889 [I] Checking connection
2026-09-28 14:49:12,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:49:13,191 [I] connection success
2026-09-28 14:49:13,191 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:49:13,191 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:49:13,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:49:13,434 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:49:13,434 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:49:13,435 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:49:13,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:50685/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:49:13,733 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:49:14.227382 18818 mini_ranger_kms.cc:310] Created kms service
I20260928 14:49:14.436573 18818 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:49:14.484740 18818 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:49:14 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606951, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:49:14.927160 18818 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:49:14.927292 18818 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:49:14.927320 18818 mini_ranger_kms.cc:240] Using host: 127.18.96.148
I20260928 14:49:14.931115 18818 mini_ranger_kms.cc:276] Ranger KMS PID: 29928
I20260928 14:49:14.931259 18818 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.96.148:53643
Sep 28, 2026 2:49:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:49:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:49:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:49:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:49:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:49:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:49:17.197259 18818 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.927s user 0.000s sys 0.447s
I20260928 14:49:17.197381 18818 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:49:17.197474 18818 mini_ranger_kms.cc:401] 127.18.96.148:53643/kms/v1/keys
Sep 28 14:49:17 dist-test-slave-j968 krb5kdc[29535](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606957, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:49:17 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606957, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.148@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
Sep 28 14:49:18 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606957, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.148@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
Sep 28 14:49:20 dist-test-slave-j968 krb5kdc[29535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:49:20.708390 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35485
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:35485
--ranger_config_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:53643/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:20.793510 29979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:20.793731 29979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:20.793752 29979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:20.794970 29979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:49:20.795015 29979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:20.795035 29979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:20.795048 29979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:49:20.795060 29979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:49:20.796725 29979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.96.148:53643/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:35485
--ranger_config_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:49:20.796985 29979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:20.797256 29979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:20.802672 29985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:20.802752 29984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:20.802789 29979 server_base.cc:1034] running on GCE node
W20260928 14:49:20.802929 29987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:20.803303 29979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:20.803671 29979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:20.804834 29979 hybrid_clock.cc:648] HybridClock initialized: now 1790606960804792 us; error 57 us; skew 500 ppm
Sep 28 14:49:20 dist-test-slave-j968 krb5kdc[29535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606960, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:20.813690 29979 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:49:20.814260 29979 webserver.cc:460] Webserver started at http://127.18.96.190:46375/ using document root <none> and password file <none>
I20260928 14:49:20.814538 29979 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:20.814604 29979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:20.814790 29979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:49:20 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606960, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:49:20.878986 29979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a"
format_stamp: "Formatted at 2026-09-28 14:49:20 on dist-test-slave-j968"
server_key: "7a1a6d770a92196e05ffd04e4eba0635"
server_key_iv: "544e0bbdc411262a0a0b9b0a5021a6fe"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:20.879467 29979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a"
format_stamp: "Formatted at 2026-09-28 14:49:20 on dist-test-slave-j968"
server_key: "7a1a6d770a92196e05ffd04e4eba0635"
server_key_iv: "544e0bbdc411262a0a0b9b0a5021a6fe"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:20.881320 29979 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 14:49:20.897536 29994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:20.897970 29979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:49:20.898352 29979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a"
format_stamp: "Formatted at 2026-09-28 14:49:20 on dist-test-slave-j968"
server_key: "7a1a6d770a92196e05ffd04e4eba0635"
server_key_iv: "544e0bbdc411262a0a0b9b0a5021a6fe"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:20.898495 29979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:20.919409 29979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:20.921799 29979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:20.921963 29979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:20.927376 29979 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:35485
I20260928 14:49:20.927428 30046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:35485 every 8 connection(s)
I20260928 14:49:20.927894 29979 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:49:20.928646 30047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:20.931084 30047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a: Bootstrap starting.
I20260928 14:49:20.931739 30047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:20.932091 30047 log.cc:824] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:20.932818 30047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a: No bootstrap required, opened a new log
I20260928 14:49:20.935190 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 29979
I20260928 14:49:20.935307 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:49:20.935187 30047 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35485 } }
I20260928 14:49:20.935434 30047 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:20.935497 30047 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbecfb2e53ba4689875dd4fb5c8c6b6a, State: Initialized, Role: FOLLOWER
I20260928 14:49:20.935544 18818 external_mini_cluster.cc:1428] Setting key 5030475d20b833442fd5fa6464902c1f
I20260928 14:49:20.935670 30047 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35485 } }
I20260928 14:49:20.935770 30047 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:49:20.935835 30047 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:49:20.935920 30047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:20.936776 30047 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35485 } }
I20260928 14:49:20.936904 30047 leader_election.cc:302] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbecfb2e53ba4689875dd4fb5c8c6b6a; no voters:
I20260928 14:49:20.937090 30047 leader_election.cc:288] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:49:20.937119 30052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:49:20.937379 30047 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:49:20.937379 30052 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [term 1 LEADER]: Becoming Leader. State: Replica: cbecfb2e53ba4689875dd4fb5c8c6b6a, State: Running, Role: LEADER
I20260928 14:49:20.937639 30052 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35485 } }
I20260928 14:49:20.938023 30054 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbecfb2e53ba4689875dd4fb5c8c6b6a. Latest consensus state: current_term: 1 leader_uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35485 } } }
I20260928 14:49:20.938098 30054 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [sys.catalog]: This master's current role is: LEADER
I20260928 14:49:20.938349 30053 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbecfb2e53ba4689875dd4fb5c8c6b6a" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35485 } } }
I20260928 14:49:20.938462 30053 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a [sys.catalog]: This master's current role is: LEADER
I20260928 14:49:20.938798 30060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:49:20.939360 30060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:49:20.940223 30047 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:49:20.944976 30060 catalog_manager.cc:1347] Generated new cluster ID: 7b824d5008174179aa27d4515f0cd02b
I20260928 14:49:20.945115 30060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:49:20 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:49:20.951254 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:20.936104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.18.96.190:35485)
0928 14:49:20.936343 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:20.936355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:20.936395 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:49:20.936500 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:20.936506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:20.936551 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:20.936676 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:20.937609 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:20.939307 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:20.940341 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:20.943411 (+ 3070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:20.944332 (+ 921us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:20.946921 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:20.946926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:20.946926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:20.946939 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:20.949089 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:20.949512 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:20.949515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:20.949516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:20.949645 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:20.949851 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:20.949852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:20.949852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:20.950177 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:20.950775 (+ 598us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:20.950852 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 14:49:20.950997 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":142,"threads_started":1}
I20260928 14:49:20.953195 30060 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:49:20.953758 30060 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:49:20.960216 30060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a: Generated new TSK 0
I20260928 14:49:20.960412 30060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:49:21.719703 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35485
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:21.832314 30092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:21.832621 30092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:21.832659 30092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:21.834439 30092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:21.834506 30092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:21.834548 30092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:49:21.836803 30092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.96.148:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:49:21.837224 30092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:21.837538 30092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:49:21.844408 30092 server_base.cc:1034] running on GCE node
W20260928 14:49:21.844331 30100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:21.844635 30097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:21.844576 30098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:21.845077 30092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:21.845417 30092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:21.846604 30092 hybrid_clock.cc:648] HybridClock initialized: now 1790606961846577 us; error 44 us; skew 500 ppm
Sep 28 14:49:21 dist-test-slave-j968 krb5kdc[29535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606961, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:21.850278 30092 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:49:21.850817 30092 webserver.cc:460] Webserver started at http://127.18.96.129:33285/ using document root <none> and password file <none>
I20260928 14:49:21.851114 30092 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:21.851163 30092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:21.851295 30092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:49:21 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606961, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:49:21.871531 30092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "1be8969a5a05481bbc46abfbbdd52888"
format_stamp: "Formatted at 2026-09-28 14:49:21 on dist-test-slave-j968"
server_key: "d2e3095ab2e875849e0d8a802f022781"
server_key_iv: "62d362f723a1d3fd745a5a27b652e879"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:21.872040 30092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "1be8969a5a05481bbc46abfbbdd52888"
format_stamp: "Formatted at 2026-09-28 14:49:21 on dist-test-slave-j968"
server_key: "d2e3095ab2e875849e0d8a802f022781"
server_key_iv: "62d362f723a1d3fd745a5a27b652e879"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:21.875176 30092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:49:21.887815 30109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:21.888096 30092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:49:21.888203 30092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "1be8969a5a05481bbc46abfbbdd52888"
format_stamp: "Formatted at 2026-09-28 14:49:21 on dist-test-slave-j968"
server_key: "d2e3095ab2e875849e0d8a802f022781"
server_key_iv: "62d362f723a1d3fd745a5a27b652e879"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:21.888296 30092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:21.909650 30092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:21.911849 30092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:21.912055 30092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:21.912295 30092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:21.912701 30092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:49:21.912755 30092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:21.912822 30092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:21.912866 30092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:21.921046 30092 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:42021
I20260928 14:49:21.921535 30092 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:49:21.930394 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 30092
I20260928 14:49:21.930505 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:49:21.930729 18818 external_mini_cluster.cc:1428] Setting key f8c9237098c25faeb427a0aa05280dab
I20260928 14:49:21.932940 30222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:42021 every 8 connection(s)
Sep 28 14:49:21 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606961, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
I20260928 14:49:21.950274 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:21.922480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35485 (local address 127.18.96.129:58153)
0928 14:49:21.928006 (+ 5526us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:21.928078 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:21.928754 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:21.933560 (+ 4806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:21.933566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:21.933838 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:21.934125 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:21.934132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:21.935298 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:21.935299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:21.936752 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:21.936755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:21.937301 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:21.937302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:21.937469 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:21.938134 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:21.938153 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:21.939878 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:21.947917 (+ 8039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:21.947930 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:21.947933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:21.948522 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:21.948855 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:21.948856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:21.948857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:21.949095 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:21.949412 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:21.949446 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:21.949651 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 14:49:21.949763 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:49:21.949959 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5450,"thread_start_us":113,"threads_started":1}
I20260928 14:49:21.950695 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:21.922495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:58153 (local address 127.18.96.190:35485)
0928 14:49:21.922666 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:21.922668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:21.933280 (+ 10612us) server_negotiation.cc:407] Connection header received
0928 14:49:21.933314 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:21.933315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:21.933329 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:21.933406 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:21.934244 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:21.935187 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:21.936854 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:21.937169 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:21.937231 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:21.939995 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:21.939997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:21.939998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:21.940005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:21.947760 (+ 7755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:21.948660 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:21.948662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:21.948662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:21.948741 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:21.949187 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:21.949188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:21.949188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:21.949335 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:21.949408 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:21.950450 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0928 14:49:21.950552 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":104,"threads_started":1}
I20260928 14:49:21.952097 30223 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35485
I20260928 14:49:21.952214 30223 heartbeater.cc:464] Registering TS with master...
I20260928 14:49:21.952415 30223 heartbeater.cc:511] Master 127.18.96.190:35485 requested a full tablet report, sending...
I20260928 14:49:21.952929 30011 ts_manager.cc:194] Registered new tserver with Master: 1be8969a5a05481bbc46abfbbdd52888 (127.18.96.129:42021)
I20260928 14:49:21.953691 30011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:58153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:49:21.971493 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35485
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:22.080001 30229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:22.080245 30229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:22.080286 30229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:22.081642 30229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:22.081712 30229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:22.081758 30229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:49:22.083813 30229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.96.148:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:49:22.084131 30229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:22.084389 30229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:22.089776 30237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:22.089749 30239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:22.089761 30236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:22.090375 30229 server_base.cc:1034] running on GCE node
I20260928 14:49:22.090581 30229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:22.090859 30229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:22.092078 30229 hybrid_clock.cc:648] HybridClock initialized: now 1790606962092057 us; error 37 us; skew 500 ppm
Sep 28 14:49:22 dist-test-slave-j968 krb5kdc[29535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:22.094864 30229 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:49:22.095342 30229 webserver.cc:460] Webserver started at http://127.18.96.130:37963/ using document root <none> and password file <none>
I20260928 14:49:22.095607 30229 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:22.095675 30229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:22.095810 30229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:49:22 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
Sep 28 14:49:22 dist-test-slave-j968 krb5kdc[29535](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:22.107365 30229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "82ce479b05bb441eb85b1a6806741992"
format_stamp: "Formatted at 2026-09-28 14:49:22 on dist-test-slave-j968"
server_key: "ad3d2b7306efc75e515d9c77e8837fe4"
server_key_iv: "8f8fb46119df7764f25acf871ed0bebd"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:22.107822 30229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "82ce479b05bb441eb85b1a6806741992"
format_stamp: "Formatted at 2026-09-28 14:49:22 on dist-test-slave-j968"
server_key: "ad3d2b7306efc75e515d9c77e8837fe4"
server_key_iv: "8f8fb46119df7764f25acf871ed0bebd"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:22.116379 30229 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
I20260928 14:49:22.136031 30246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:22.136351 30229 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.001s
I20260928 14:49:22.136461 30229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "82ce479b05bb441eb85b1a6806741992"
format_stamp: "Formatted at 2026-09-28 14:49:22 on dist-test-slave-j968"
server_key: "ad3d2b7306efc75e515d9c77e8837fe4"
server_key_iv: "8f8fb46119df7764f25acf871ed0bebd"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:22.136570 30229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:22.151224 30229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:22.153671 30229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:22.153909 30229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:22.154210 30229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:22.154603 30229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:49:22.154667 30229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:22.154736 30229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:22.154765 30229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:22.162819 30229 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:38015
I20260928 14:49:22.162858 30359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:38015 every 8 connection(s)
I20260928 14:49:22.163268 30229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:49:22 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:49:22.173671 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 30229
I20260928 14:49:22.173776 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:49:22.173981 18818 external_mini_cluster.cc:1428] Setting key 871701592cc5ed747b77b65dc2a955ce
I20260928 14:49:22.180687 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.164332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:39761 (local address 127.18.96.190:35485)
0928 14:49:22.164413 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:22.164415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:22.168152 (+ 3737us) server_negotiation.cc:407] Connection header received
0928 14:49:22.168715 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:22.168716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:22.168733 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:22.168783 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:22.169433 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.169969 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.171030 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.171238 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.171317 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.176430 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:22.176432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:22.176433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:22.176443 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:22.178505 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:22.179197 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:22.179198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:22.179199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:22.179271 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:22.179620 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:22.179621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:22.179622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:22.179788 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:22.179890 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:22.180300 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 14:49:22.180412 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:49:22.180727 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.164105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35485 (local address 127.18.96.130:39761)
0928 14:49:22.167991 (+ 3886us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:22.168040 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:22.168601 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:22.168856 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:22.168860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:22.169072 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:22.169334 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.169339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.170068 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.170069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.170920 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.170922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.171323 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.171324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.171484 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.172098 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:22.172107 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:22.176292 (+ 4185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:22.178628 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:22.178637 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:22.178640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:22.179049 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:22.179361 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:22.179361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:22.179362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:22.179528 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:22.179905 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:22.179912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:22.180121 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 14:49:22.180262 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 14:49:22.180454 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3834,"thread_start_us":364,"threads_started":1}
I20260928 14:49:22.181654 30360 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35485
I20260928 14:49:22.181777 30360 heartbeater.cc:464] Registering TS with master...
I20260928 14:49:22.182010 30360 heartbeater.cc:511] Master 127.18.96.190:35485 requested a full tablet report, sending...
I20260928 14:49:22.182382 30011 ts_manager.cc:194] Registered new tserver with Master: 82ce479b05bb441eb85b1a6806741992 (127.18.96.130:38015)
I20260928 14:49:22.182899 30011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:39761
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:49:22.203310 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35485
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:22.309799 30366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:22.310077 30366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:22.310123 30366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:22.311493 30366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:22.311612 30366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:22.311672 30366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:49:22.313587 30366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.96.148:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:49:22.313879 30366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:22.314158 30366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:22.319806 30372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:22.320039 30371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:22.320149 30374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:22.321193 30366 server_base.cc:1034] running on GCE node
I20260928 14:49:22.321420 30366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:22.321988 30366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:22.323197 30366 hybrid_clock.cc:648] HybridClock initialized: now 1790606962323171 us; error 45 us; skew 500 ppm
Sep 28 14:49:22 dist-test-slave-j968 krb5kdc[29535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:22.326563 30366 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:49:22.327203 30366 webserver.cc:460] Webserver started at http://127.18.96.131:41695/ using document root <none> and password file <none>
I20260928 14:49:22.327521 30366 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:22.327608 30366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:22.327819 30366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:49:22 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:49:22.341631 30366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "1a58e1aff0cd4e60b1d07afcb911421f"
format_stamp: "Formatted at 2026-09-28 14:49:22 on dist-test-slave-j968"
server_key: "1807375b6cc4ed74a951b338586b6fd5"
server_key_iv: "01833c64c4bb60ae78b01fa1a16ff2d0"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:22.342482 30366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "1a58e1aff0cd4e60b1d07afcb911421f"
format_stamp: "Formatted at 2026-09-28 14:49:22 on dist-test-slave-j968"
server_key: "1807375b6cc4ed74a951b338586b6fd5"
server_key_iv: "01833c64c4bb60ae78b01fa1a16ff2d0"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:22.347424 30366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260928 14:49:22.367550 30381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:22.367820 30366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:49:22.367969 30366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "1a58e1aff0cd4e60b1d07afcb911421f"
format_stamp: "Formatted at 2026-09-28 14:49:22 on dist-test-slave-j968"
server_key: "1807375b6cc4ed74a951b338586b6fd5"
server_key_iv: "01833c64c4bb60ae78b01fa1a16ff2d0"
server_key_version: "kuduclusterkey@0"
I20260928 14:49:22.368065 30366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:22.401181 30366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:22.403553 30366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:22.403729 30366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:22.403993 30366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:22.404385 30366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:49:22.404446 30366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:22.404489 30366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:22.404515 30366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:22.411783 30366 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:40783
I20260928 14:49:22.411906 30495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:40783 every 8 connection(s)
I20260928 14:49:22.412223 30366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:49:22 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:49:22.421511 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 30366
I20260928 14:49:22.421638 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:49:22.421820 18818 external_mini_cluster.cc:1428] Setting key 322d1d7146eec75e837b9912724145ff
I20260928 14:49:22.423022 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.413115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:37775 (local address 127.18.96.190:35485)
0928 14:49:22.413161 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:22.413164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:22.413377 (+ 213us) server_negotiation.cc:407] Connection header received
0928 14:49:22.413858 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:22.413859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:22.413872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:22.413941 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:22.414490 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.415270 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.416456 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.416799 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.416890 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.419265 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:22.419267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:22.419267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:22.419274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:22.420869 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:22.422010 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:22.422011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:22.422011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:22.422065 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:22.422348 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:22.422349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:22.422349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:22.422478 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:22.422549 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:22.422846 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 14:49:22.422929 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:49:22.423241 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.413028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35485 (local address 127.18.96.131:37775)
0928 14:49:22.413242 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:22.413278 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:22.413755 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:22.414004 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:22.414007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:22.414191 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:22.414399 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.414403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.415362 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.415363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.416378 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.416380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.416941 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.416943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.417052 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.417718 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:22.417727 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:22.419138 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:22.421050 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:22.421056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:22.421058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:22.421278 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:22.422148 (+ 870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:22.422150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:22.422150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:22.422272 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:22.422552 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:22.422559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:22.422710 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 14:49:22.422814 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:49:22.422978 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260928 14:49:22.424610 30496 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35485
I20260928 14:49:22.424736 30496 heartbeater.cc:464] Registering TS with master...
I20260928 14:49:22.424923 30496 heartbeater.cc:511] Master 127.18.96.190:35485 requested a full tablet report, sending...
I20260928 14:49:22.425427 30010 ts_manager.cc:194] Registered new tserver with Master: 1a58e1aff0cd4e60b1d07afcb911421f (127.18.96.131:40783)
I20260928 14:49:22.425943 30010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:37775
I20260928 14:49:22.426401 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:49:22.443779 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.428062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.18.96.190:35485)
0928 14:49:22.428101 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:22.428103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:22.435725 (+ 7622us) server_negotiation.cc:407] Connection header received
0928 14:49:22.435840 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:22.435841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:22.435853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:22.435909 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:22.436751 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.437461 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.438511 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.438803 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.438848 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.440614 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:22.440616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:22.440616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:22.440638 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:22.442483 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:22.442894 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:22.442896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:22.442896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:22.442949 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:22.443150 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:22.443151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:22.443151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:22.443280 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:22.443355 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:22.443595 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 14:49:22.443675 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:49:22.446408 30010 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:49:22.448731 30010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:49:22.456936 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.451611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:40783 (local address 127.0.0.1:43460)
0928 14:49:22.452011 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:22.452055 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:22.452120 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:22.452521 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:22.452525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:22.452527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:22.452600 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.452603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.454165 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.454166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.455503 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.455506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.456637 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.456638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.456718 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.456731 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 14:49:22.456802 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:49:22.456855 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":72,"threads_started":1}
I20260928 14:49:22.457388 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.452042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.18.96.131:40783)
0928 14:49:22.452269 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:22.452274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:22.452284 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:49:22.452323 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:22.452326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:22.452343 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:22.452413 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:22.452953 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.454070 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.455628 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.456454 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.456526 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.456603 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:22.457255 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 14:49:22.457305 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
I20260928 14:49:22.458178 30430 tablet_service.cc:1467] Processing CreateTablet for tablet ad192431569148a9b8f64408d7bbb871 (DEFAULT_TABLE table=test-table [id=00652c1c22a240dbb98ea4cecf7fcea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:49:22.458604 30430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad192431569148a9b8f64408d7bbb871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:22.460764 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.451842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:38015 (local address 127.0.0.1:58884)
0928 14:49:22.452865 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:22.452874 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:22.452908 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:22.454481 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:22.454482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:22.454485 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:22.454803 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.454806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.458220 (+ 3414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.458221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.459410 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.459413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.460526 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.460527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.460588 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.460592 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:49:22.460643 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:49:22.460682 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":91,"threads_started":1}
I20260928 14:49:22.461066 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.451617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:42021 (local address 127.0.0.1:51974)
0928 14:49:22.452723 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:22.452734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:22.452786 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:22.456275 (+ 3489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:22.456276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:22.456278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:22.456340 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.456341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.458791 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.458792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.459908 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.459910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.460850 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.460852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.460928 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.460932 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:49:22.460966 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 14:49:22.461003 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":82,"threads_started":1}
I20260928 14:49:22.461092 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.452328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.18.96.130:38015)
0928 14:49:22.454236 (+ 1908us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:22.454240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:22.454250 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:49:22.454290 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:22.454293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:22.454309 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:22.454387 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:22.454886 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.455965 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.459547 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.460356 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.460422 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.460492 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:22.460965 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 14:49:22.461007 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1870,"thread_start_us":83,"threads_started":1}
I20260928 14:49:22.461732 30294 tablet_service.cc:1467] Processing CreateTablet for tablet ad192431569148a9b8f64408d7bbb871 (DEFAULT_TABLE table=test-table [id=00652c1c22a240dbb98ea4cecf7fcea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:49:22.462134 30294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad192431569148a9b8f64408d7bbb871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:22.463217 30515 tablet_bootstrap.cc:492] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: Bootstrap starting.
I20260928 14:49:22.463644 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.452505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.18.96.129:42021)
0928 14:49:22.456099 (+ 3594us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:22.456104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:22.456113 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:49:22.456149 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:22.456151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:22.456168 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:22.456421 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:22.456501 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.458696 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.460020 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.460755 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.463381 (+ 2626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.463463 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:22.463493 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 14:49:22.463547 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3559,"thread_start_us":50,"threads_started":1}
I20260928 14:49:22.464344 30515 tablet_bootstrap.cc:654] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:22.465029 30515 log.cc:824] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:22.465041 30157 tablet_service.cc:1467] Processing CreateTablet for tablet ad192431569148a9b8f64408d7bbb871 (DEFAULT_TABLE table=test-table [id=00652c1c22a240dbb98ea4cecf7fcea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:49:22.465373 30157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad192431569148a9b8f64408d7bbb871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:22.465809 30515 tablet_bootstrap.cc:492] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: No bootstrap required, opened a new log
I20260928 14:49:22.465869 30515 ts_tablet_manager.cc:1397] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 14:49:22.467736 30518 tablet_bootstrap.cc:492] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888: Bootstrap starting.
I20260928 14:49:22.467849 30515 raft_consensus.cc:348] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } }
I20260928 14:49:22.468050 30515 raft_consensus.cc:374] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:22.468096 30515 raft_consensus.cc:729] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a58e1aff0cd4e60b1d07afcb911421f, State: Initialized, Role: FOLLOWER
I20260928 14:49:22.468215 30515 consensus_queue.cc:260] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } }
I20260928 14:49:22.468500 30515 ts_tablet_manager.cc:1428] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:49:22.468739 30496 heartbeater.cc:503] Master 127.18.96.190:35485 was elected leader, sending a full tablet report...
I20260928 14:49:22.468757 30518 tablet_bootstrap.cc:654] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:22.469147 30518 log.cc:824] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:22.469916 30518 tablet_bootstrap.cc:492] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888: No bootstrap required, opened a new log
I20260928 14:49:22.469981 30518 ts_tablet_manager.cc:1397] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:49:22.472599 30518 raft_consensus.cc:348] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } }
I20260928 14:49:22.472754 30518 raft_consensus.cc:374] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:22.472783 30518 raft_consensus.cc:729] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be8969a5a05481bbc46abfbbdd52888, State: Initialized, Role: FOLLOWER
I20260928 14:49:22.472900 30518 consensus_queue.cc:260] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } }
I20260928 14:49:22.473057 30517 tablet_bootstrap.cc:492] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: Bootstrap starting.
I20260928 14:49:22.473186 30518 ts_tablet_manager.cc:1428] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:22.473423 30223 heartbeater.cc:503] Master 127.18.96.190:35485 was elected leader, sending a full tablet report...
I20260928 14:49:22.474941 30517 tablet_bootstrap.cc:654] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:22.475258 30517 log.cc:824] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:22.476174 30517 tablet_bootstrap.cc:492] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: No bootstrap required, opened a new log
I20260928 14:49:22.476298 30517 ts_tablet_manager.cc:1397] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:49:22.478374 30517 raft_consensus.cc:348] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } }
I20260928 14:49:22.478591 30517 raft_consensus.cc:374] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:22.478647 30517 raft_consensus.cc:729] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ce479b05bb441eb85b1a6806741992, State: Initialized, Role: FOLLOWER
I20260928 14:49:22.478771 30517 consensus_queue.cc:260] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } }
I20260928 14:49:22.479084 30360 heartbeater.cc:503] Master 127.18.96.190:35485 was elected leader, sending a full tablet report...
I20260928 14:49:22.479318 30517 ts_tablet_manager.cc:1428] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
Sep 28 14:49:22 dist-test-slave-j968 krb5kdc[29535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:49:22.607353 30523 raft_consensus.cc:482] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:49:22.607512 30523 raft_consensus.cc:504] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } }
I20260928 14:49:22.607967 30523 leader_election.cc:288] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a58e1aff0cd4e60b1d07afcb911421f (127.18.96.131:40783), 1be8969a5a05481bbc46abfbbdd52888 (127.18.96.129:42021)
I20260928 14:49:22.612681 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.608399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:37037 (local address 127.18.96.131:40783)
0928 14:49:22.608619 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:22.608621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:22.609459 (+ 838us) server_negotiation.cc:407] Connection header received
0928 14:49:22.609562 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:22.609564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:22.609573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:22.609602 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:22.609871 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.610531 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.611770 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.612332 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.612377 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.612386 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:22.612575 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:49:22.612614 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260928 14:49:22.612694 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.608114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:40783 (local address 127.18.96.130:37037)
0928 14:49:22.609385 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:22.609396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:22.609486 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:22.609664 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:22.609665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:22.609667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:22.609782 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.609784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.610655 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.610656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.611673 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.611676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.612442 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.612443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.612500 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.612503 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:49:22.612570 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:49:22.612609 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1251,"mutex_wait_us":54,"thread_start_us":106,"threads_started":1}
I20260928 14:49:22.616348 30450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad192431569148a9b8f64408d7bbb871" candidate_uuid: "82ce479b05bb441eb85b1a6806741992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" is_pre_election: true
I20260928 14:49:22.616559 30450 raft_consensus.cc:2454] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82ce479b05bb441eb85b1a6806741992 in term 0.
I20260928 14:49:22.616813 30247 leader_election.cc:302] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a58e1aff0cd4e60b1d07afcb911421f, 82ce479b05bb441eb85b1a6806741992; no voters:
I20260928 14:49:22.616945 30523 raft_consensus.cc:2781] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:49:22.616984 30523 raft_consensus.cc:482] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:49:22.617005 30523 raft_consensus.cc:3037] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:22.617861 30523 raft_consensus.cc:504] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } }
I20260928 14:49:22.617991 30523 leader_election.cc:288] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [CANDIDATE]: Term 1 election: Requested vote from peers 1a58e1aff0cd4e60b1d07afcb911421f (127.18.96.131:40783), 1be8969a5a05481bbc46abfbbdd52888 (127.18.96.129:42021)
I20260928 14:49:22.618229 30450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad192431569148a9b8f64408d7bbb871" candidate_uuid: "82ce479b05bb441eb85b1a6806741992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a58e1aff0cd4e60b1d07afcb911421f"
I20260928 14:49:22.618307 30450 raft_consensus.cc:3037] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:22.619117 30450 raft_consensus.cc:2454] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82ce479b05bb441eb85b1a6806741992 in term 1.
I20260928 14:49:22.619282 30247 leader_election.cc:302] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a58e1aff0cd4e60b1d07afcb911421f, 82ce479b05bb441eb85b1a6806741992; no voters:
I20260928 14:49:22.619364 30523 raft_consensus.cc:2781] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:49:22.619506 30523 raft_consensus.cc:686] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 1 LEADER]: Becoming Leader. State: Replica: 82ce479b05bb441eb85b1a6806741992, State: Running, Role: LEADER
I20260928 14:49:22.619625 30523 consensus_queue.cc:237] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } }
I20260928 14:49:22.620414 30011 catalog_manager.cc:5495] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82ce479b05bb441eb85b1a6806741992 (127.18.96.130). New cstate: current_term: 1 leader_uuid: "82ce479b05bb441eb85b1a6806741992" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82ce479b05bb441eb85b1a6806741992" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 38015 } health_report { overall_health: HEALTHY } } }
I20260928 14:49:22.620532 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.608114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:42021 (local address 127.18.96.130:50983)
0928 14:49:22.608213 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:22.608225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:22.608265 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:22.608569 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:22.608569 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:22.608572 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:22.608642 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.608644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.609566 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.609567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.610790 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:22.610793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.620232 (+ 9439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.620233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:22.620318 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.620323 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:49:22.620379 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:49:22.620432 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":1}
I20260928 14:49:22.620759 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.608350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:50983 (local address 127.18.96.129:42021)
0928 14:49:22.608441 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:22.608443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:22.608451 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:49:22.608481 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:22.608482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:22.608492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:22.608561 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:22.608708 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.609463 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.616211 (+ 6748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.616855 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.616929 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.616939 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:22.620630 (+ 3691us) server_negotiation.cc:299] Negotiation successful
0928 14:49:22.620693 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:49:22.621194 30177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad192431569148a9b8f64408d7bbb871" candidate_uuid: "82ce479b05bb441eb85b1a6806741992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be8969a5a05481bbc46abfbbdd52888" is_pre_election: true
I20260928 14:49:22.621340 30177 raft_consensus.cc:2454] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82ce479b05bb441eb85b1a6806741992 in term 0.
I20260928 14:49:22.621440 30176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad192431569148a9b8f64408d7bbb871" candidate_uuid: "82ce479b05bb441eb85b1a6806741992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be8969a5a05481bbc46abfbbdd52888"
I20260928 14:49:22.621482 30176 raft_consensus.cc:3037] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:22.622306 30176 raft_consensus.cc:2454] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82ce479b05bb441eb85b1a6806741992 in term 1.
I20260928 14:49:22.640414 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.636954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58888 (local address 127.18.96.130:38015)
0928 14:49:22.637001 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:22.637003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:22.637012 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:49:22.637061 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:22.637062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:22.637080 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:22.637176 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:49:22.637358 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.638111 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.639349 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:22.639667 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:22.639737 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:22.639968 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:49:22.640129 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:49:22.640212 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:22.640302 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 14:49:22.640329 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:49:22.642730 30176 raft_consensus.cc:1261] T ad192431569148a9b8f64408d7bbb871 P 1be8969a5a05481bbc46abfbbdd52888 [term 1 FOLLOWER]: Refusing update from remote peer 82ce479b05bb441eb85b1a6806741992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:49:22.643188 30450 raft_consensus.cc:1261] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [term 1 FOLLOWER]: Refusing update from remote peer 82ce479b05bb441eb85b1a6806741992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:49:22.643528 30525 consensus_queue.cc:1035] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a58e1aff0cd4e60b1d07afcb911421f" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 40783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:49:22.643828 30525 consensus_queue.cc:1035] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be8969a5a05481bbc46abfbbdd52888" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 42021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:49:22.649873 30529 mvcc.cc:205] Tried to move back new op lower bound from 7334326118981898240 to 7334326118890672128. Current Snapshot: MvccSnapshot[applied={T|T < 7334326118981898240 or (T in {7334326118981898240})}]
I20260928 14:49:22.650004 30528 mvcc.cc:205] Tried to move back new op lower bound from 7334326118981898240 to 7334326118890672128. Current Snapshot: MvccSnapshot[applied={T|T < 7334326118981898240}]
I20260928 14:49:22.651358 30531 mvcc.cc:205] Tried to move back new op lower bound from 7334326118981898240 to 7334326118890672128. Current Snapshot: MvccSnapshot[applied={T|T < 7334326118981898240}]
I20260928 14:49:22.658301 30011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:49:22.658401 30011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:49:22.659184 30011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbecfb2e53ba4689875dd4fb5c8c6b6a: Sending DeleteTablet for 3 replicas of tablet ad192431569148a9b8f64408d7bbb871
I20260928 14:49:22.659700 30430 tablet_service.cc:1514] Processing DeleteTablet for tablet ad192431569148a9b8f64408d7bbb871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:49:22 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:43460
I20260928 14:49:22.659940 30294 tablet_service.cc:1514] Processing DeleteTablet for tablet ad192431569148a9b8f64408d7bbb871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:49:22 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:58884
I20260928 14:49:22.660140 30157 tablet_service.cc:1514] Processing DeleteTablet for tablet ad192431569148a9b8f64408d7bbb871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:49:22 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:51974
I20260928 14:49:22.660817 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 30092
I20260928 14:49:22.662540 30537 tablet_replica.cc:330] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: stopping tablet replica
I20260928 14:49:22.662657 30537 raft_consensus.cc:2229] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:49:22.662770 30537 raft_consensus.cc:2258] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:49:22.664309 30537 ts_tablet_manager.cc:1905] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:49:22.666528 30537 ts_tablet_manager.cc:1918] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:49:22.666672 30537 log.cc:1196] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/ad192431569148a9b8f64408d7bbb871
I20260928 14:49:22.667049 30537 ts_tablet_manager.cc:1939] T ad192431569148a9b8f64408d7bbb871 P 1a58e1aff0cd4e60b1d07afcb911421f: Deleting consensus metadata
I20260928 14:49:22.667565 29995 catalog_manager.cc:4841] TS 1a58e1aff0cd4e60b1d07afcb911421f (127.18.96.131:40783): tablet ad192431569148a9b8f64408d7bbb871 (table test-table [id=00652c1c22a240dbb98ea4cecf7fcea3]) successfully deleted
I20260928 14:49:22.675981 30538 tablet_replica.cc:330] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: stopping tablet replica
I20260928 14:49:22.676225 30538 raft_consensus.cc:2229] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:49:22.676468 30538 raft_consensus.cc:2258] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:49:22.679638 29997 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:42021 (error 108)
I20260928 14:49:22.680421 30538 ts_tablet_manager.cc:1905] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:49:22.682243 30538 ts_tablet_manager.cc:1918] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:49:22.682407 30538 log.cc:1196] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/ad192431569148a9b8f64408d7bbb871
I20260928 14:49:22.682720 30538 ts_tablet_manager.cc:1939] T ad192431569148a9b8f64408d7bbb871 P 82ce479b05bb441eb85b1a6806741992: Deleting consensus metadata
I20260928 14:49:22.683204 29996 catalog_manager.cc:4841] TS 82ce479b05bb441eb85b1a6806741992 (127.18.96.130:38015): tablet ad192431569148a9b8f64408d7bbb871 (table test-table [id=00652c1c22a240dbb98ea4cecf7fcea3]) successfully deleted
I20260928 14:49:22.683813 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 30229
I20260928 14:49:22.688344 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:22.688209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:42021 (local address 127.0.0.1:51986)
0928 14:49:22.688266 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:22.688307 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:49:22.688568 29997 catalog_manager.cc:4584] TS 1be8969a5a05481bbc46abfbbdd52888 (127.18.96.129:42021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad192431569148a9b8f64408d7bbb871: Network error: Client connection negotiation failed: client connection to 127.18.96.129:42021: connect: Connection refused (error 111)
I20260928 14:49:22.688787 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 30366
I20260928 14:49:22.694347 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 29979
I20260928 14:49:22.699519 18818 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:49:23.044624 18818 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:49:23.044729 18818 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:49:23.389685 18818 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:49:23.390 UTC [29565] LOG: received smart shutdown request
2026-09-28 14:49:23.393 UTC [29565] LOG: background worker "logical replication launcher" (PID 29574) exited with exit code 1
2026-09-28 14:49:23.393 UTC [29569] LOG: shutting down
2026-09-28 14:49:23.579 UTC [29565] LOG: database system is shut down
2026-09-28T14:49:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47778 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:49:23 dist-test-slave-j968 krb5kdc[30549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:49:23 dist-test-slave-j968 krb5kdc[30549](info): set up 2 sockets
Sep 28 14:49:23 dist-test-slave-j968 krb5kdc[30549](info): commencing operation
krb5kdc: starting...
W20260928 14:49:24.744923 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:49:24 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:49:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.96.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.96.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/rangeradmin_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/rangeradmin_127.18.96.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.96.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.96.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/rangerlookup_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/rangerlookup_127.18.96.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.96.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/HTTP_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/HTTP_127.18.96.148@KRBTEST.COM.keytab.
I20260928 14:49:24.930353 18818 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklohQF_/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/postgres -l logfile start
2026-09-28 14:49:27.593 UTC [30579] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:49:27.593 UTC [30579] LOG: listening on IPv4 address "127.18.96.148", port 45815
2026-09-28 14:49:27.596 UTC [30579] LOG: listening on Unix socket "/tmp/.s.PGSQL.45815"
2026-09-28 14:49:27.618 UTC [30582] LOG: database system was shut down at 2026-09-28 14:49:26 UTC
2026-09-28 14:49:27.622 UTC [30579] LOG: database system is ready to accept connections
I20260928 14:49:28.633280 18818 mini_postgres.cc:96] Postgres bound to 45815
2026-09-28 14:49:28.639 UTC [30590] FATAL: database "slave" does not exist
127.18.96.148:45815 - accepting connections
I20260928 14:49:28.639948 18818 mini_ranger.cc:161] Starting Ranger...
I20260928 14:49:28.656919 18818 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:49:29.200541 18818 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:49:29.200640 18818 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-admin
2026-09-28 14:49:29,495 [I] DB FLAVOR :POSTGRES
2026-09-28 14:49:29,495 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:49:29,495 [I] Checking connection..
2026-09-28 14:49:29,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:49:29,748 [I] Checking connection passed.
2026-09-28 14:49:29,748 [I] --------- Verifying version history table ---------
2026-09-28 14:49:29,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:49:29,991 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:49:29,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:49:30,231 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:49:30,231 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:49:30,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:49:30.465 UTC [30654] WARNING: there is no transaction in progress
2026-09-28 14:49:30,472 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:49:30,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:49:30,737 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:49:30,737 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:49:30,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:49:30,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:49:31,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j968.c.gcp-upstream.internal','N') ;"
2026-09-28 14:49:31,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:49:31,759 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:49:31,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:49:32,004 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:49:32,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:49:32,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:49:32,493 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:49:32,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:49:33.558 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.576 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.594 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.610 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.627 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.648 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.717 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.725 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.742 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.752 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.763 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.778 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.789 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.797 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.804 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:33.811 UTC [30763] WARNING: there is no transaction in progress
2026-09-28 14:49:34,485 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:49:34,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:49:34,725 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:49:34,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:49:34,959 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:49:34,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:49:35,195 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:49:35,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:49:35,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:49:35,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j968.c.gcp-upstream.internal';"
2026-09-28 14:49:35,927 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:49:35,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:49:36,176 [I] DB_PATCHES have already been applied
I20260928 14:49:36.182190 18818 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklohQF_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:49:36.182256 18818 mini_ranger.cc:193] Using host: 127.18.96.148
I20260928 14:49:36.185464 18818 mini_ranger.cc:227] Ranger admin URL: http://127.18.96.148:42117
Sep 28, 2026 2:49:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:49:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:49:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:49:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:49:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:49:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:49:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:49:37.310555 18818 mini_ranger.cc:160] Time spent starting Ranger: real 8.671s user 0.000s sys 0.010s
Sep 28, 2026 2:49:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:49:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:49:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:49:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:49:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:49:59.291422 18818 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.96.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.96.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/rangerkms_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.96.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/rangerkms_127.18.96.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.96.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.96.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/HTTP_127.18.96.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.96.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/HTTP_127.18.96.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:49:59 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606999, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:49:59.385792 18818 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:49:59.395841 18818 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:50:00.375833 18818 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:50:00.375973 18818 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-kms
I20260928 14:50:00.375993 18818 mini_ranger_kms.cc:227] Using postgres at 127.18.96.148:45815
2026-09-28 14:50:00,788 [I] DB FLAVOR :POSTGRES
2026-09-28 14:50:00,789 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:50:00,789 [I] Checking connection
2026-09-28 14:50:00,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:50:01,061 [I] connection success
2026-09-28 14:50:01,061 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:50:01,061 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:50:01,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:50:01,353 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:50:01,354 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:50:01,354 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:50:01,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.148:45815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:50:01,662 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:50:02.166703 18818 mini_ranger_kms.cc:310] Created kms service
I20260928 14:50:02.400588 18818 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:50:02.446774 18818 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:50:02 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606999, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:02.889626 18818 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:50:02.889847 18818 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklohQF_/build/release/bin/postgresql.jar:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklohQF_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:50:02.889869 18818 mini_ranger_kms.cc:240] Using host: 127.18.96.148
I20260928 14:50:02.893322 18818 mini_ranger_kms.cc:276] Ranger KMS PID: 30945
I20260928 14:50:02.893409 18818 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.96.148:32935
Sep 28, 2026 2:50:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:50:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:50:05.127168 18818 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.741s user 0.002s sys 0.172s
I20260928 14:50:05.127276 18818 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:50:05.127336 18818 mini_ranger_kms.cc:401] 127.18.96.148:32935/kms/v1/keys
Sep 28 14:50:05 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607005, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:50:05 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607005, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.148@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
Sep 28 14:50:06 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607005, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.148@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
Sep 28 14:50:08 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:50:08.310799 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43821
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:43821
--ranger_config_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:32935/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:08.409338 30996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:08.409583 30996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:08.409618 30996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:08.410787 30996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:08.410851 30996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:08.410872 30996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:08.410892 30996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:08.410919 30996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:08.412500 30996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.96.148:32935/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:43821
--ranger_config_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:08.412751 30996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:08.412978 30996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:08.418417 31002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:08.418439 31001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:08.418459 30996 server_base.cc:1034] running on GCE node
W20260928 14:50:08.418694 31004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:08.418978 30996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:08.419317 30996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:08.420490 30996 hybrid_clock.cc:648] HybridClock initialized: now 1790607008420455 us; error 50 us; skew 500 ppm
Sep 28 14:50:08 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607008, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:08.427523 30996 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:50:08.428071 30996 webserver.cc:460] Webserver started at http://127.18.96.190:42963/ using document root <none> and password file <none>
I20260928 14:50:08.428332 30996 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:08.428398 30996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:08.428547 30996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:50:08 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607008, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:08.489776 30996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "52573585e61d46bca7bb3e4f30f17313"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-j968"
server_key: "8b23aceeefd86cfcd3e24c3cd8ecf24b"
server_key_iv: "82285e38767cf290faed477b179d9221"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:08.490195 30996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "52573585e61d46bca7bb3e4f30f17313"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-j968"
server_key: "8b23aceeefd86cfcd3e24c3cd8ecf24b"
server_key_iv: "82285e38767cf290faed477b179d9221"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:08.491660 30996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:50:08.507287 31011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:08.507560 30996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:50:08.507684 30996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "52573585e61d46bca7bb3e4f30f17313"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-j968"
server_key: "8b23aceeefd86cfcd3e24c3cd8ecf24b"
server_key_iv: "82285e38767cf290faed477b179d9221"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:08.507788 30996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:08.535434 30996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:08.537719 30996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:08.537904 30996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:08.543525 30996 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:43821
I20260928 14:50:08.543605 31063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:43821 every 8 connection(s)
I20260928 14:50:08.544059 30996 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:08.544919 31064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:08.547222 31064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Bootstrap starting.
I20260928 14:50:08.547705 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 30996
I20260928 14:50:08.547825 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:50:08.547915 31064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:08.548064 18818 external_mini_cluster.cc:1428] Setting key a10986c4c5f246d6f9c86616f2c6d861
I20260928 14:50:08.548297 31064 log.cc:824] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:08.549105 31064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: No bootstrap required, opened a new log
I20260928 14:50:08.551627 31064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } }
I20260928 14:50:08.551798 31064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:08.551859 31064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52573585e61d46bca7bb3e4f30f17313, State: Initialized, Role: FOLLOWER
I20260928 14:50:08.552052 31064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } }
I20260928 14:50:08.552145 31064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:08.552207 31064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:08.552266 31064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:08.553048 31064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } }
I20260928 14:50:08.553157 31064 leader_election.cc:302] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52573585e61d46bca7bb3e4f30f17313; no voters:
I20260928 14:50:08.553328 31064 leader_election.cc:288] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:08.553390 31070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:08.553610 31070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 1 LEADER]: Becoming Leader. State: Replica: 52573585e61d46bca7bb3e4f30f17313, State: Running, Role: LEADER
I20260928 14:50:08.553714 31064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:08.553745 31070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } }
I20260928 14:50:08.554286 31070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52573585e61d46bca7bb3e4f30f17313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } } }
I20260928 14:50:08.554419 31070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:08.554590 31070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52573585e61d46bca7bb3e4f30f17313. Latest consensus state: current_term: 1 leader_uuid: "52573585e61d46bca7bb3e4f30f17313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } } }
I20260928 14:50:08.554685 31070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:08.555351 31064 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 14:50:08 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:08.558391 31079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:08.559523 31079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:08.563040 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.548699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.18.96.190:43821)
0928 14:50:08.548849 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.548860 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.548899 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:50:08.549605 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.549613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.549709 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.549875 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:08.550685 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.552240 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.553575 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.554211 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.554307 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.557910 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:08.557980 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:08.557980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:08.558104 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:08.561049 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:08.561502 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:08.561506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:08.561506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:08.561618 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:08.561861 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:08.561862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:08.561862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:08.562212 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:08.562354 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.562632 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.562811 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260928 14:50:08.567469 31079 catalog_manager.cc:1347] Generated new cluster ID: a71eaad087da49e3ac95ed76b3cd5e47
I20260928 14:50:08.567559 31079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:08.577809 31079 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:08.578653 31079 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:08.587669 31079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Generated new TSK 0
I20260928 14:50:08.588003 31079 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:50:09.254858 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:32935/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43821
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:09.376183 31110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:09.376549 31110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:09.376608 31110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:09.378332 31110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:09.378397 31110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:09.378444 31110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:50:09.380687 31110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.96.148:32935/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:09.380939 31110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:09.381206 31110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:09.387616 31115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:09.387580 31116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:09.387993 31118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:09.387993 31110 server_base.cc:1034] running on GCE node
I20260928 14:50:09.388245 31110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:09.388489 31110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:09.389645 31110 hybrid_clock.cc:648] HybridClock initialized: now 1790607009389631 us; error 30 us; skew 500 ppm
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:09.392545 31110 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:50:09.393065 31110 webserver.cc:460] Webserver started at http://127.18.96.129:38337/ using document root <none> and password file <none>
I20260928 14:50:09.393312 31110 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:09.393357 31110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:09.393493 31110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:09.408805 31110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "badc2934947c4fcc87d9afae8a509e7c"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "bb669334a927d1222122e4c7a51fbe4f"
server_key_iv: "6a50001488e9a75410bc88f4a72e2fb1"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:09.409403 31110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "badc2934947c4fcc87d9afae8a509e7c"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "bb669334a927d1222122e4c7a51fbe4f"
server_key_iv: "6a50001488e9a75410bc88f4a72e2fb1"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:09.411300 31110 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:50:09.428125 31126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.430167 31110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:50:09.430303 31110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "badc2934947c4fcc87d9afae8a509e7c"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "bb669334a927d1222122e4c7a51fbe4f"
server_key_iv: "6a50001488e9a75410bc88f4a72e2fb1"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:09.430403 31110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:09.442287 31110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:09.444728 31110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:09.444895 31110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:09.445200 31110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:09.445600 31110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:09.445701 31110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.446010 31110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:09.446106 31110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.454170 31110 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:34735
I20260928 14:50:09.454216 31241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:34735 every 8 connection(s)
I20260928 14:50:09.454763 31110 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:09.462321 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 31110
I20260928 14:50:09.462442 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:50:09.462641 18818 external_mini_cluster.cc:1428] Setting key 914cb91e830dfb080b08ceed8f359465
I20260928 14:50:09.466250 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.455995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:50839 (local address 127.18.96.190:43821)
0928 14:50:09.456212 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.456214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.456225 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:50:09.456704 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.456706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.456720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.456816 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:09.457537 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.458466 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.459473 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.459770 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.459849 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.462547 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:09.462548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:09.462549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:09.462557 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:09.464666 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:09.465190 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:09.465191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:09.465192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:09.465281 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:09.465543 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:09.465544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:09.465544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:09.465668 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:09.465743 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.466041 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.466143 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":133,"threads_started":1}
I20260928 14:50:09.466678 31244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.455594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43821 (local address 127.18.96.129:50839)
0928 14:50:09.455810 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.455850 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.456549 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.456896 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.456900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.457154 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:09.457431 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.457437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.458567 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.458567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.459384 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.459386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.459906 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.459907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.460047 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.460677 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:09.460688 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:09.462390 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:09.464761 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:09.464769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:09.464771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:09.465109 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:09.465348 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:09.465349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:09.465349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:09.465477 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:09.465733 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:09.465740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:09.465891 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.466009 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.466223 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":117,"threads_started":1}
I20260928 14:50:09.467145 31242 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43821
I20260928 14:50:09.467257 31242 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:09.467446 31242 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
I20260928 14:50:09.468101 31028 ts_manager.cc:194] Registered new tserver with Master: badc2934947c4fcc87d9afae8a509e7c (127.18.96.129:34735)
I20260928 14:50:09.469017 31028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:50839
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:50:09.493510 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:32935/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43821
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 14:50:09.612993 31248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:09.613333 31248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:09.613380 31248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:09.615182 31248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:09.615247 31248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:09.615302 31248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:50:09.617561 31248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.96.148:32935/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:09.617831 31248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:09.618096 31248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:09.624799 31257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:09.624887 31259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:09.625377 31256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:09.625440 31248 server_base.cc:1034] running on GCE node
I20260928 14:50:09.625681 31248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:09.625958 31248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:09.627130 31248 hybrid_clock.cc:648] HybridClock initialized: now 1790607009627105 us; error 35 us; skew 500 ppm
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:09.630414 31248 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:50:09.630944 31248 webserver.cc:460] Webserver started at http://127.18.96.130:36669/ using document root <none> and password file <none>
I20260928 14:50:09.631212 31248 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:09.631265 31248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:09.631394 31248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:09.651499 31248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "11812bed3e8143a69e77be20f362feb4"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "5809b309cbe4f45a9bae07a16cb31a53"
server_key_iv: "d41e30122fe6fbaac2708fb8140aa107"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:09.651980 31248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "11812bed3e8143a69e77be20f362feb4"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "5809b309cbe4f45a9bae07a16cb31a53"
server_key_iv: "d41e30122fe6fbaac2708fb8140aa107"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:09.653791 31248 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:50:09.668044 31266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.668300 31248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:09.668455 31248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "11812bed3e8143a69e77be20f362feb4"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "5809b309cbe4f45a9bae07a16cb31a53"
server_key_iv: "d41e30122fe6fbaac2708fb8140aa107"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:09.668610 31248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:09.687247 31248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:09.689702 31248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:09.689858 31248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:09.690199 31248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:09.694886 31248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:09.694944 31248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.694978 31248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:09.694996 31248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.702451 31248 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:39331
I20260928 14:50:09.702526 31381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:39331 every 8 connection(s)
I20260928 14:50:09.702862 31248 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:09.710412 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 31248
I20260928 14:50:09.710534 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:50:09.710765 18818 external_mini_cluster.cc:1428] Setting key 72239923e1cede70b1842d8b46993079
I20260928 14:50:09.713404 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.703674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:46773 (local address 127.18.96.190:43821)
0928 14:50:09.703729 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.703731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.703974 (+ 243us) server_negotiation.cc:407] Connection header received
0928 14:50:09.704607 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.704608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.704626 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.704676 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:09.705330 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.705900 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.707082 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.707293 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.707374 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.709822 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:09.709824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:09.709824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:09.709831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:09.711559 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:09.712184 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:09.712185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:09.712186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:09.712248 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:09.712534 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:09.712535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:09.712535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:09.712681 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:09.712731 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.713218 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.713316 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:50:09.714146 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.703675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43821 (local address 127.18.96.130:46773)
0928 14:50:09.703833 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.703892 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.704488 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.704745 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.704749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.704998 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:09.705199 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.705222 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.706093 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.706094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.706987 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.706990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.707400 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.707402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.707532 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.708122 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:09.708130 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:09.709709 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:09.711646 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:09.711653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:09.711656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:09.712095 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:09.712318 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:09.712319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:09.712319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:09.712463 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:09.712763 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:09.712770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:09.713033 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.713711 (+ 678us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.713890 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260928 14:50:09.715685 31382 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43821
I20260928 14:50:09.715793 31382 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:09.716126 31382 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
I20260928 14:50:09.716537 31028 ts_manager.cc:194] Registered new tserver with Master: 11812bed3e8143a69e77be20f362feb4 (127.18.96.130:39331)
I20260928 14:50:09.717044 31028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:46773
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:50:09.747328 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:32935/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43821
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:09.861816 31387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:09.862071 31387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:09.862110 31387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:09.863320 31387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:09.863386 31387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:09.863430 31387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:50:09.865008 31387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.96.148:32935/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:09.865281 31387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:09.865523 31387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:09.872709 31393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:09.872720 31395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:09.872839 31392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:09.872989 31387 server_base.cc:1034] running on GCE node
I20260928 14:50:09.873291 31387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:09.873605 31387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:09.874787 31387 hybrid_clock.cc:648] HybridClock initialized: now 1790607009874766 us; error 47 us; skew 500 ppm
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:09.877988 31387 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:50:09.878626 31387 webserver.cc:460] Webserver started at http://127.18.96.131:42475/ using document root <none> and password file <none>
I20260928 14:50:09.878968 31387 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:09.879093 31387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:09.879339 31387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:09.897091 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "15da2ca6043d4e5eb42d0f42a1155c34"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "40321930e9e6a789e25d25b632bd6844"
server_key_iv: "43bcdeab3a75fd9f6ca7e0b0c2a9281f"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:09.897615 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "15da2ca6043d4e5eb42d0f42a1155c34"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "40321930e9e6a789e25d25b632bd6844"
server_key_iv: "43bcdeab3a75fd9f6ca7e0b0c2a9281f"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:09.902698 31387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 14:50:09.916160 31402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.916352 31387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:09.916471 31387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "15da2ca6043d4e5eb42d0f42a1155c34"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "40321930e9e6a789e25d25b632bd6844"
server_key_iv: "43bcdeab3a75fd9f6ca7e0b0c2a9281f"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:09.916591 31387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:09.938516 31387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:09.940683 31387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:09.940848 31387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:09.941121 31387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:09.941546 31387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:09.941598 31387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.941638 31387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:09.941664 31387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.949118 31387 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:44583
I20260928 14:50:09.949239 31516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:44583 every 8 connection(s)
I20260928 14:50:09.949599 31387 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:09.955739 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 31387
I20260928 14:50:09.955884 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:50:09.956072 18818 external_mini_cluster.cc:1428] Setting key 6a18331ac3cc8da3c8770f9c1897426e
Sep 28 14:50:09 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607009, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:09.966404 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.954636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:59205 (local address 127.18.96.190:43821)
0928 14:50:09.954682 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.954684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.954979 (+ 295us) server_negotiation.cc:407] Connection header received
0928 14:50:09.955825 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.955826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.955845 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.955910 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:09.956852 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.957648 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.959437 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.959766 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.959850 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.962671 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:09.962673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:09.962673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:09.962683 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:09.964624 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:09.965224 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:09.965226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:09.965226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:09.965293 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:09.965616 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:09.965617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:09.965617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:09.965765 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:09.965832 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.966213 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.966311 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:50:09.966641 31519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.954301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43821 (local address 127.18.96.131:59205)
0928 14:50:09.954835 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.954889 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.955679 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.956143 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.956148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.956425 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:09.956730 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.956738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.957775 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.957775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.959338 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.959341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.959925 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.959926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.960090 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.960776 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:09.960788 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:09.962529 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:09.964732 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:09.964739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:09.964742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:09.965097 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:09.965377 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:09.965378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:09.965378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:09.965540 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:09.965839 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:09.965845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:09.966043 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.966183 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.966375 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":97,"threads_started":1}
I20260928 14:50:09.967424 31517 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43821
I20260928 14:50:09.967535 31517 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:09.968003 31517 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
I20260928 14:50:09.968410 31028 ts_manager.cc:194] Registered new tserver with Master: 15da2ca6043d4e5eb42d0f42a1155c34 (127.18.96.131:44583)
I20260928 14:50:09.968729 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:09.968859 31028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:59205
I20260928 14:50:09.969417 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 31110
I20260928 14:50:09.977306 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 31248
I20260928 14:50:09.984477 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 31387
I20260928 14:50:09.991037 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 30996
I20260928 14:50:09.996685 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43821
--webserver_interface=127.18.96.190
--webserver_port=42963
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:43821
--ranger_config_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:32935/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.96.148:32935/kms with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:10.080669 31520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:10.080920 31520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:10.080984 31520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:10.082159 31520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:10.082201 31520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:10.082216 31520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:10.082252 31520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:10.082283 31520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:10.083880 31520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.96.148:32935/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:43821
--ranger_config_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:10.084151 31520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:10.084400 31520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:10.089356 31525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:10.089385 31526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:10.089393 31528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:10.089429 31520 server_base.cc:1034] running on GCE node
I20260928 14:50:10.089722 31520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:10.090032 31520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:10.091207 31520 hybrid_clock.cc:648] HybridClock initialized: now 1790607010091189 us; error 28 us; skew 500 ppm
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:10.093657 31520 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:50:10.094069 31520 webserver.cc:460] Webserver started at http://127.18.96.190:42963/ using document root <none> and password file <none>
I20260928 14:50:10.094305 31520 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:10.094373 31520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:10.152369 31520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:10.153223 31536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.153438 31520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:10.153568 31520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "52573585e61d46bca7bb3e4f30f17313"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-j968"
server_key: "8b23aceeefd86cfcd3e24c3cd8ecf24b"
server_key_iv: "82285e38767cf290faed477b179d9221"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:10.154043 31520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:10.175959 31520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:10.177752 31520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:10.177939 31520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:10.182809 31520 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:43821
I20260928 14:50:10.182848 31588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:43821 every 8 connection(s)
I20260928 14:50:10.183257 31520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:10.184556 31589 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:50:10.185201 31589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Bootstrap starting.
I20260928 14:50:10.192662 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 31520
I20260928 14:50:10.192966 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:32935/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:34735
--local_ip_for_outbound_sockets=127.18.96.129
--tserver_master_addrs=127.18.96.190:43821
--webserver_port=38337
--webserver_interface=127.18.96.129
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.96.148:32935/kms with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:10.193810 31589 log.cc:824] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:10.195547 31589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:50:10.195835 31589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Bootstrap complete.
I20260928 14:50:10.198346 31589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } }
I20260928 14:50:10.198637 31589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52573585e61d46bca7bb3e4f30f17313, State: Initialized, Role: FOLLOWER
I20260928 14:50:10.198838 31589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } }
I20260928 14:50:10.198930 31589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:10.198982 31589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:10.199030 31589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:50:10.200187 31589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } }
I20260928 14:50:10.200306 31589 leader_election.cc:302] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52573585e61d46bca7bb3e4f30f17313; no voters:
I20260928 14:50:10.200500 31589 leader_election.cc:288] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:50:10.200536 31593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:50:10.200637 31593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [term 2 LEADER]: Becoming Leader. State: Replica: 52573585e61d46bca7bb3e4f30f17313, State: Running, Role: LEADER
I20260928 14:50:10.200744 31593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } }
I20260928 14:50:10.200773 31589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:10.201228 31595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52573585e61d46bca7bb3e4f30f17313. Latest consensus state: current_term: 2 leader_uuid: "52573585e61d46bca7bb3e4f30f17313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } } }
I20260928 14:50:10.201304 31595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:10.201557 31597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:10.202080 31597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:10.202219 31597 catalog_manager.cc:1259] Loaded cluster ID: a71eaad087da49e3ac95ed76b3cd5e47
I20260928 14:50:10.202248 31597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:10.202674 31589 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:50:10.203332 31594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "52573585e61d46bca7bb3e4f30f17313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52573585e61d46bca7bb3e4f30f17313" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43821 } } }
I20260928 14:50:10.203459 31594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:10.203507 31597 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:10.203685 31597 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Loaded TSK: 0
I20260928 14:50:10.203931 31597 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:50:10.281507 31591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:10.281739 31591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:10.281778 31591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:10.283469 31591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:10.283528 31591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:10.283569 31591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:50:10.286019 31591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.96.148:32935/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:34735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=38337
--tserver_master_addrs=127.18.96.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:10.286262 31591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:10.286514 31591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:10.293596 31626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:10.294121 31627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:10.294298 31629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:10.294903 31591 server_base.cc:1034] running on GCE node
I20260928 14:50:10.295130 31591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:10.295415 31591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:10.296591 31591 hybrid_clock.cc:648] HybridClock initialized: now 1790607010296556 us; error 42 us; skew 500 ppm
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:10.299974 31591 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:50:10.300572 31591 webserver.cc:460] Webserver started at http://127.18.96.129:38337/ using document root <none> and password file <none>
I20260928 14:50:10.300976 31591 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:10.301095 31591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:10.324067 31591 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:10.325275 31637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.325527 31591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:10.325665 31591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "badc2934947c4fcc87d9afae8a509e7c"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "bb669334a927d1222122e4c7a51fbe4f"
server_key_iv: "6a50001488e9a75410bc88f4a72e2fb1"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:10.326223 31591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:10.349040 31591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:10.351644 31591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:10.351902 31591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:10.352242 31591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:10.352655 31591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:10.352739 31591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.352841 31591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:10.352917 31591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.359459 31591 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:34735
I20260928 14:50:10.360021 31591 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:10.365211 31752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:34735 every 8 connection(s)
I20260928 14:50:10.369705 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 31591
I20260928 14:50:10.370028 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:32935/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:39331
--local_ip_for_outbound_sockets=127.18.96.130
--tserver_master_addrs=127.18.96.190:43821
--webserver_port=36669
--webserver_interface=127.18.96.130
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.96.148:32935/kms with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:10.378811 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:10.365934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43821 (local address 127.18.96.129:43971)
0928 14:50:10.366158 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:10.366217 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:10.366827 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:10.367350 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:10.367355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:10.367599 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:10.367906 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:10.367913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.369916 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.369917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:10.371334 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:10.371337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.372285 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.372286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:10.372419 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:10.373024 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:10.373034 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:10.374652 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:10.377011 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:10.377017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:10.377019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:10.377318 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:10.377591 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:10.377592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:10.377592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:10.377704 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:10.378146 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:10.378152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:10.378299 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 14:50:10.378408 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:50:10.378578 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":106,"threads_started":1}
I20260928 14:50:10.379396 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:10.366341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:43971 (local address 127.18.96.190:43821)
0928 14:50:10.366736 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:10.366745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:10.366786 (+ 41us) server_negotiation.cc:407] Connection header received
0928 14:50:10.367072 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:10.367079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:10.367138 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:10.367285 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:10.368043 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.369703 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.371434 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.372074 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.372176 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:10.374789 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:10.374794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:10.374794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:10.374806 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:10.376915 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:10.377413 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:10.377416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:10.377416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:10.377526 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:10.377770 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:10.377771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:10.377771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:10.378037 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:10.378116 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:10.379039 (+ 923us) server_negotiation.cc:299] Negotiation successful
0928 14:50:10.379164 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":86,"threads_started":1}
I20260928 14:50:10.380101 31753 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43821
I20260928 14:50:10.380200 31753 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:10.380390 31753 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
W20260928 14:50:10.381199 31753 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:43821 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:50:10.381276 31753 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:10.381351 31753 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
I20260928 14:50:10.381652 31753 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:10.381727 31753 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
W20260928 14:50:10.382006 31753 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:50:10.491374 31757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:10.491663 31757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:10.491789 31757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:10.493736 31757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:10.493811 31757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:10.493857 31757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:50:10.496383 31757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.96.148:32935/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:39331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=36669
--tserver_master_addrs=127.18.96.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:10.496668 31757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:10.496970 31757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:10.503582 31765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:10.503746 31757 server_base.cc:1034] running on GCE node
W20260928 14:50:10.503433 31762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:10.503428 31763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:10.504073 31757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:10.504374 31757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:10.505538 31757 hybrid_clock.cc:648] HybridClock initialized: now 1790607010505511 us; error 47 us; skew 500 ppm
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:10.509167 31757 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:50:10.509750 31757 webserver.cc:460] Webserver started at http://127.18.96.130:36669/ using document root <none> and password file <none>
I20260928 14:50:10.510056 31757 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:10.510128 31757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:10.526619 31757 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:10.527781 31773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.527994 31757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:10.528097 31757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "11812bed3e8143a69e77be20f362feb4"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "5809b309cbe4f45a9bae07a16cb31a53"
server_key_iv: "d41e30122fe6fbaac2708fb8140aa107"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:10.528611 31757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:10.550104 31757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:10.552594 31757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:10.552752 31757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:10.553016 31757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:10.553337 31757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:10.553376 31757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.553411 31757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:10.553431 31757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.559608 31757 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:39331
I20260928 14:50:10.560132 31757 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:10.568120 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 31757
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:10.568475 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.148:32935/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:44583
--local_ip_for_outbound_sockets=127.18.96.131
--tserver_master_addrs=127.18.96.190:43821
--webserver_port=42475
--webserver_interface=127.18.96.131
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.96.148:32935/kms with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:10.572279 31887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:39331 every 8 connection(s)
I20260928 14:50:10.572934 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:10.561082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:50967 (local address 127.18.96.190:43821)
0928 14:50:10.561341 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:10.561343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:10.561458 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:50:10.562208 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:10.562210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:10.562226 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:10.562297 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:10.563067 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.563856 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.565432 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.565764 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.565846 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:10.569187 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:10.569190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:10.569190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:10.569198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:10.570969 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:10.571585 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:10.571587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:10.571587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:10.571654 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:10.572030 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:10.572032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:10.572032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:10.572180 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:10.572227 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:10.572732 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 14:50:10.572833 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260928 14:50:10.573771 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:10.561083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43821 (local address 127.18.96.130:50967)
0928 14:50:10.561292 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:10.561344 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:10.562071 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:10.562397 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:10.562402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:10.562656 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:10.562952 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:10.562959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.564006 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.564007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:10.565332 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:10.565335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.565939 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.565940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:10.566100 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:10.566759 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:10.566770 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:10.569052 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:10.571078 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:10.571088 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:10.571092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:10.571478 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:10.571750 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:10.571751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:10.571751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:10.571938 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:10.572280 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:10.572287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:10.572553 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 14:50:10.573312 (+ 759us) client_negotiation.cc:240] Negotiation successful
0928 14:50:10.573522 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260928 14:50:10.574489 31888 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43821
I20260928 14:50:10.574587 31888 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:10.574788 31888 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
W20260928 14:50:10.575428 31888 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:43821 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:50:10.575518 31888 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:10.575589 31888 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
I20260928 14:50:10.575913 31888 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:10.575978 31888 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
W20260928 14:50:10.576237 31888 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:50:10.698246 31891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:10.698553 31891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:10.698597 31891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:10.700431 31891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:10.700502 31891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:10.700549 31891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:50:10.702960 31891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.96.148:32935/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:44583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=42475
--tserver_master_addrs=127.18.96.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:10.703236 31891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:10.703519 31891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:10.710125 31897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:10.710246 31899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:10.711568 31891 server_base.cc:1034] running on GCE node
W20260928 14:50:10.711722 31896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:10.717672 31891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:10.718041 31891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:10.721433 31891 hybrid_clock.cc:648] HybridClock initialized: now 1790607010721396 us; error 51 us; skew 500 ppm
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:10.724709 31891 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:50:10.725310 31891 webserver.cc:460] Webserver started at http://127.18.96.131:42475/ using document root <none> and password file <none>
I20260928 14:50:10.725596 31891 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:10.725673 31891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:10.739595 31891 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:10.740794 31908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.741012 31891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:50:10.741125 31891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "15da2ca6043d4e5eb42d0f42a1155c34"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-j968"
server_key: "40321930e9e6a789e25d25b632bd6844"
server_key_iv: "43bcdeab3a75fd9f6ca7e0b0c2a9281f"
server_key_version: "kuduclusterkey@0"
I20260928 14:50:10.741626 31891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:10.758926 31891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:10.760900 31891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:10.761045 31891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:10.761286 31891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:10.761606 31891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:10.761644 31891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.761673 31891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:10.761691 31891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.768589 31891 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:44583
I20260928 14:50:10.768693 32022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:44583 every 8 connection(s)
I20260928 14:50:10.769037 31891 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:10.770763 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 31891
Sep 28 14:50:10 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:10.782019 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:10.769822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:50317 (local address 127.18.96.190:43821)
0928 14:50:10.769878 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:10.769881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:10.770116 (+ 235us) server_negotiation.cc:407] Connection header received
0928 14:50:10.770690 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:10.770691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:10.770705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:10.770769 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:10.771688 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.772487 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.773670 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.773961 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.774034 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:10.778558 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:10.778560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:10.778560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:10.778568 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:10.780273 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:10.780838 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:10.780839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:10.780839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:10.780899 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:10.781229 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:10.781230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:10.781230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:10.781374 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:10.781426 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:10.781828 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 14:50:10.781911 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:50:10.782229 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:10.769822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43821 (local address 127.18.96.131:50317)
0928 14:50:10.769993 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:10.770031 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:10.770562 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:10.770866 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:10.770872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:10.771108 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:10.771577 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:10.771584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.772591 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.772591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:10.773584 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:10.773586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.774135 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.774137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:10.774280 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:10.775079 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:10.775103 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:10.778436 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:10.780388 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:10.780397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:10.780399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:10.780734 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:10.780974 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:10.780974 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:10.780975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:10.781137 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:10.781479 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:10.781485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:10.781677 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 14:50:10.781808 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 14:50:10.781984 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260928 14:50:10.782760 32023 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43821
I20260928 14:50:10.782862 32023 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:10.783056 32023 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
W20260928 14:50:10.783710 32023 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:43821 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:50:10.783792 32023 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:10.783901 32023 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
I20260928 14:50:10.784221 32023 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:10.784299 32023 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
W20260928 14:50:10.784600 32023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:50:10.785251 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:10.771210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.18.96.190:43821)
0928 14:50:10.771374 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:10.771376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:10.771431 (+ 55us) server_negotiation.cc:407] Connection header received
0928 14:50:10.771507 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:10.771507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:10.771519 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:10.771593 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:10.772354 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.773072 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.774393 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:10.774679 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:10.774747 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:10.776177 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:10.776178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:10.776179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:10.776186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:10.777953 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:10.783925 (+ 5972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:10.783926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:10.783927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:10.784034 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:10.784601 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:10.784601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:10.784602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:10.784810 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:10.785000 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:10.785048 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 14:50:10.785137 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":54,"threads_started":1}
Sep 28 14:50:11 dist-test-slave-j968 krb5kdc[30549](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790607011, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:11.382193 31753 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:11.382371 31753 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
I20260928 14:50:11.382809 31553 ts_manager.cc:194] Registered new tserver with Master: badc2934947c4fcc87d9afae8a509e7c (127.18.96.129:34735)
I20260928 14:50:11.383616 31553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:43971
I20260928 14:50:11.576403 31888 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:11.576550 31888 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
I20260928 14:50:11.576901 31553 ts_manager.cc:194] Registered new tserver with Master: 11812bed3e8143a69e77be20f362feb4 (127.18.96.130:39331)
I20260928 14:50:11.577443 31553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:50967
Sep 28 14:50:11 dist-test-slave-j968 krb5kdc[30549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607011, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for HTTP/127.18.96.148@KRBTEST.COM
I20260928 14:50:11.784782 32023 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:11.784945 32023 heartbeater.cc:511] Master 127.18.96.190:43821 requested a full tablet report, sending...
I20260928 14:50:11.785333 31553 ts_manager.cc:194] Registered new tserver with Master: 15da2ca6043d4e5eb42d0f42a1155c34 (127.18.96.131:44583)
I20260928 14:50:11.785876 31553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:50317
I20260928 14:50:11.787294 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:11.801705 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.788297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.18.96.190:43821)
0928 14:50:11.788555 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.788557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.788711 (+ 154us) server_negotiation.cc:407] Connection header received
0928 14:50:11.788733 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.788734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.788745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.788809 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:11.790624 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.791391 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.792493 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.792818 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.792874 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.798359 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:11.798360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:11.798361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:11.798369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:11.800265 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.800707 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.800709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.800709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.800769 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.800949 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.800949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.800950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.801089 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:11.801181 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.801466 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.801598 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":61,"threads_started":1}
I20260928 14:50:11.804402 31553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:11.806627 31553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:11.820327 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.810182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:44583 (local address 127.0.0.1:44216)
0928 14:50:11.811262 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.811300 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.815023 (+ 3723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.815352 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.815355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.815357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:11.815435 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.815436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.817248 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.817249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.818588 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.818590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.819675 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.819676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.820065 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.820091 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.820155 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.820213 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":64,"threads_started":1}
I20260928 14:50:11.820793 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.810750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.18.96.131:44583)
0928 14:50:11.811199 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.811204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.815133 (+ 3929us) server_negotiation.cc:407] Connection header received
0928 14:50:11.815175 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.815178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.815196 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.815260 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:11.815523 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.816647 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.818699 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.819477 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.819546 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.819605 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.820664 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.820710 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":78,"threads_started":1}
I20260928 14:50:11.821775 31956 tablet_service.cc:1467] Processing CreateTablet for tablet 357eef3e7ef74e449abda204d6f63ae3 (DEFAULT_TABLE table=test-table [id=d043d3df7b554528aba1beedf7ad3226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:11.822184 31956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357eef3e7ef74e449abda204d6f63ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:11.823892 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.810544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39331 (local address 127.0.0.1:42974)
0928 14:50:11.816897 (+ 6353us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.816904 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.816935 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.820220 (+ 3285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.820221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.820223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:11.820257 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.820258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.821485 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.821486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.822613 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.822615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.823646 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.823647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.823705 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.823710 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.823761 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.823802 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6342,"thread_start_us":59,"threads_started":1}
I20260928 14:50:11.824196 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.810650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.18.96.130:39331)
0928 14:50:11.819971 (+ 9321us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.819976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.819989 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:50:11.820025 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.820027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.820045 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.820129 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:11.820333 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.821384 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.822709 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.823487 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.823557 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.823613 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.824070 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.824117 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9267,"thread_start_us":40,"threads_started":1}
I20260928 14:50:11.824829 31821 tablet_service.cc:1467] Processing CreateTablet for tablet 357eef3e7ef74e449abda204d6f63ae3 (DEFAULT_TABLE table=test-table [id=d043d3df7b554528aba1beedf7ad3226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:11.825173 31821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357eef3e7ef74e449abda204d6f63ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:11.831147 32058 tablet_bootstrap.cc:492] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: Bootstrap starting.
I20260928 14:50:11.831539 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.810397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:34735 (local address 127.0.0.1:35248)
0928 14:50:11.816777 (+ 6380us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.816788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.816828 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.827829 (+ 11001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.827830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.827833 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:11.827908 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.827909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.829227 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.829228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.830368 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.830370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.831323 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.831324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.831394 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.831399 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.831427 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.831465 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6357,"thread_start_us":68,"threads_started":1}
I20260928 14:50:11.831856 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.810510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.18.96.129:34735)
0928 14:50:11.827658 (+ 17148us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.827663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.827675 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:50:11.827709 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.827711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.827728 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.827997 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:11.828076 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.829126 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.830471 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.831219 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.831647 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.831716 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.831742 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.831779 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17107,"thread_start_us":52,"threads_started":1}
I20260928 14:50:11.832513 31685 tablet_service.cc:1467] Processing CreateTablet for tablet 357eef3e7ef74e449abda204d6f63ae3 (DEFAULT_TABLE table=test-table [id=d043d3df7b554528aba1beedf7ad3226]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:11.832836 31685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357eef3e7ef74e449abda204d6f63ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:11.836908 32058 tablet_bootstrap.cc:654] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:11.837307 32058 log.cc:824] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:11.837708 32057 tablet_bootstrap.cc:492] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: Bootstrap starting.
I20260928 14:50:11.838430 32057 tablet_bootstrap.cc:654] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:11.838811 32057 log.cc:824] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:11.839184 32059 tablet_bootstrap.cc:492] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c: Bootstrap starting.
I20260928 14:50:11.839809 32059 tablet_bootstrap.cc:654] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:11.840204 32059 log.cc:824] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:11.840884 32058 tablet_bootstrap.cc:492] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: No bootstrap required, opened a new log
I20260928 14:50:11.841003 32058 ts_tablet_manager.cc:1397] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 14:50:11.843684 32058 raft_consensus.cc:348] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } }
I20260928 14:50:11.844008 32058 raft_consensus.cc:374] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:11.844115 32058 raft_consensus.cc:729] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11812bed3e8143a69e77be20f362feb4, State: Initialized, Role: FOLLOWER
I20260928 14:50:11.844323 32058 consensus_queue.cc:260] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } }
I20260928 14:50:11.844682 32058 ts_tablet_manager.cc:1428] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 14:50:11.844861 31888 heartbeater.cc:503] Master 127.18.96.190:43821 was elected leader, sending a full tablet report...
I20260928 14:50:11.846727 32057 tablet_bootstrap.cc:492] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: No bootstrap required, opened a new log
I20260928 14:50:11.846864 32057 ts_tablet_manager.cc:1397] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 14:50:11.849637 32057 raft_consensus.cc:348] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } }
I20260928 14:50:11.849879 32057 raft_consensus.cc:374] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:11.849974 32057 raft_consensus.cc:729] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15da2ca6043d4e5eb42d0f42a1155c34, State: Initialized, Role: FOLLOWER
I20260928 14:50:11.850188 32057 consensus_queue.cc:260] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } }
I20260928 14:50:11.850555 32057 ts_tablet_manager.cc:1428] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 14:50:11.850721 32023 heartbeater.cc:503] Master 127.18.96.190:43821 was elected leader, sending a full tablet report...
I20260928 14:50:11.852784 32059 tablet_bootstrap.cc:492] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c: No bootstrap required, opened a new log
I20260928 14:50:11.852913 32059 ts_tablet_manager.cc:1397] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260928 14:50:11.855656 32059 raft_consensus.cc:348] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } }
I20260928 14:50:11.855930 32059 raft_consensus.cc:374] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:11.856029 32059 raft_consensus.cc:729] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badc2934947c4fcc87d9afae8a509e7c, State: Initialized, Role: FOLLOWER
I20260928 14:50:11.856223 32059 consensus_queue.cc:260] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } }
I20260928 14:50:11.856617 32059 ts_tablet_manager.cc:1428] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 14:50:11.856827 31753 heartbeater.cc:503] Master 127.18.96.190:43821 was elected leader, sending a full tablet report...
W20260928 14:50:11.867663 31754 tablet.cc:2369] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:11.868805 32064 raft_consensus.cc:482] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:11.869028 32064 raft_consensus.cc:504] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } }
I20260928 14:50:11.870349 32064 leader_election.cc:288] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers badc2934947c4fcc87d9afae8a509e7c (127.18.96.129:34735), 11812bed3e8143a69e77be20f362feb4 (127.18.96.130:39331)
I20260928 14:50:11.876650 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.869899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:48565 (local address 127.18.96.129:34735)
0928 14:50:11.869944 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.869947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.870587 (+ 640us) server_negotiation.cc:407] Connection header received
0928 14:50:11.870699 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.870700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.870711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.870740 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:11.870972 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.871708 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.875641 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.876220 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.876273 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.876282 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.876528 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.876573 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:50:11.877089 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.869697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:34735 (local address 127.18.96.131:48565)
0928 14:50:11.870519 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.870531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.870619 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.870814 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.870814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.870816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:11.870894 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.870896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.871811 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.871812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.873066 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.873069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.876365 (+ 3296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.876366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.876436 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.876439 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.876944 (+ 505us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.876990 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":103,"threads_started":1}
I20260928 14:50:11.877564 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357eef3e7ef74e449abda204d6f63ae3" candidate_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badc2934947c4fcc87d9afae8a509e7c" is_pre_election: true
I20260928 14:50:11.877743 31706 raft_consensus.cc:2454] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15da2ca6043d4e5eb42d0f42a1155c34 in term 0.
I20260928 14:50:11.878064 31911 leader_election.cc:302] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15da2ca6043d4e5eb42d0f42a1155c34, badc2934947c4fcc87d9afae8a509e7c; no voters:
I20260928 14:50:11.878269 32064 raft_consensus.cc:2781] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:11.878322 32064 raft_consensus.cc:482] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:11.878345 32064 raft_consensus.cc:3037] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:11.879261 32064 raft_consensus.cc:504] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } }
I20260928 14:50:11.879386 32064 leader_election.cc:288] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [CANDIDATE]: Term 1 election: Requested vote from peers badc2934947c4fcc87d9afae8a509e7c (127.18.96.129:34735), 11812bed3e8143a69e77be20f362feb4 (127.18.96.130:39331)
I20260928 14:50:11.879695 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357eef3e7ef74e449abda204d6f63ae3" candidate_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badc2934947c4fcc87d9afae8a509e7c"
I20260928 14:50:11.879772 31706 raft_consensus.cc:3037] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:11.880716 31706 raft_consensus.cc:2454] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15da2ca6043d4e5eb42d0f42a1155c34 in term 1.
I20260928 14:50:11.880926 31911 leader_election.cc:302] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15da2ca6043d4e5eb42d0f42a1155c34, badc2934947c4fcc87d9afae8a509e7c; no voters:
I20260928 14:50:11.881040 32064 raft_consensus.cc:2781] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:11.881189 32064 raft_consensus.cc:686] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 1 LEADER]: Becoming Leader. State: Replica: 15da2ca6043d4e5eb42d0f42a1155c34, State: Running, Role: LEADER
I20260928 14:50:11.881278 32064 consensus_queue.cc:237] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } }
I20260928 14:50:11.882156 31553 catalog_manager.cc:5495] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15da2ca6043d4e5eb42d0f42a1155c34 (127.18.96.131). New cstate: current_term: 1 leader_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 44583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 } health_report { overall_health: UNKNOWN } } }
I20260928 14:50:11.886008 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.870287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:50841 (local address 127.18.96.130:39331)
0928 14:50:11.870330 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.870332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.873260 (+ 2928us) server_negotiation.cc:407] Connection header received
0928 14:50:11.873362 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.873364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.873374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.873405 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:11.873617 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.874309 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.883293 (+ 8984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.883931 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.883988 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.883998 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.885879 (+ 1881us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.885935 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:50:11.886406 32066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.870085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:39331 (local address 127.18.96.131:50841)
0928 14:50:11.873201 (+ 3116us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.873212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.873287 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.873469 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.873470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.873472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:11.873538 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.873540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.874412 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.874413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.875524 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.875527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.885694 (+ 10167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.885695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.885786 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.885789 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.886266 (+ 477us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.886320 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3100,"thread_start_us":90,"threads_started":1}
I20260928 14:50:11.886889 31841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357eef3e7ef74e449abda204d6f63ae3" candidate_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11812bed3e8143a69e77be20f362feb4" is_pre_election: true
I20260928 14:50:11.887034 31841 raft_consensus.cc:2454] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15da2ca6043d4e5eb42d0f42a1155c34 in term 0.
I20260928 14:50:11.887153 31840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357eef3e7ef74e449abda204d6f63ae3" candidate_uuid: "15da2ca6043d4e5eb42d0f42a1155c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11812bed3e8143a69e77be20f362feb4"
I20260928 14:50:11.887204 31840 raft_consensus.cc:3037] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:11.888276 31840 raft_consensus.cc:2454] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15da2ca6043d4e5eb42d0f42a1155c34 in term 1.
I20260928 14:50:11.891987 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.888691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.18.96.131:44583)
0928 14:50:11.888727 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.888730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.888738 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:50:11.888826 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.888827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.888845 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.888919 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:11.889120 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.889748 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.890947 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.891232 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.891290 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.891480 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:11.891597 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:11.891665 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.891897 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.891917 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:50:11.893749 31840 raft_consensus.cc:1261] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [term 1 FOLLOWER]: Refusing update from remote peer 15da2ca6043d4e5eb42d0f42a1155c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:11.894090 32068 consensus_queue.cc:1035] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11812bed3e8143a69e77be20f362feb4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 39331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:11.896642 31706 raft_consensus.cc:1261] T 357eef3e7ef74e449abda204d6f63ae3 P badc2934947c4fcc87d9afae8a509e7c [term 1 FOLLOWER]: Refusing update from remote peer 15da2ca6043d4e5eb42d0f42a1155c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:11.896940 32068 consensus_queue.cc:1035] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "badc2934947c4fcc87d9afae8a509e7c" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:11.900836 32071 mvcc.cc:205] Tried to move back new op lower bound from 7334326320714190848 to 7334326320666238976. Current Snapshot: MvccSnapshot[applied={T|T < 7334326320714190848}]
I20260928 14:50:11.901053 32074 mvcc.cc:205] Tried to move back new op lower bound from 7334326320714190848 to 7334326320666238976. Current Snapshot: MvccSnapshot[applied={T|T < 7334326320714190848}]
I20260928 14:50:11.903582 32072 mvcc.cc:205] Tried to move back new op lower bound from 7334326320714190848 to 7334326320666238976. Current Snapshot: MvccSnapshot[applied={T|T < 7334326320714190848}]
I20260928 14:50:11.906387 31553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:11.906528 31553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:11.907402 31553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52573585e61d46bca7bb3e4f30f17313: Sending DeleteTablet for 3 replicas of tablet 357eef3e7ef74e449abda204d6f63ae3
I20260928 14:50:11.907846 31685 tablet_service.cc:1514] Processing DeleteTablet for tablet 357eef3e7ef74e449abda204d6f63ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:11 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:35248
I20260928 14:50:11.908205 31821 tablet_service.cc:1514] Processing DeleteTablet for tablet 357eef3e7ef74e449abda204d6f63ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:11 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:42974
I20260928 14:50:11.908706 31956 tablet_service.cc:1514] Processing DeleteTablet for tablet 357eef3e7ef74e449abda204d6f63ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:11 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:44216
I20260928 14:50:11.909219 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 31591
I20260928 14:50:11.913143 32081 tablet_replica.cc:330] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: stopping tablet replica
I20260928 14:50:11.913283 32082 tablet_replica.cc:330] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: stopping tablet replica
I20260928 14:50:11.914158 32082 raft_consensus.cc:2229] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:50:11.914323 32081 raft_consensus.cc:2229] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:11.914510 32082 raft_consensus.cc:2258] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:11.914640 32081 raft_consensus.cc:2258] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:11.915299 32082 ts_tablet_manager.cc:1905] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:11.923120 32082 ts_tablet_manager.cc:1918] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:11.923646 32081 ts_tablet_manager.cc:1905] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:11.925325 32082 log.cc:1196] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/357eef3e7ef74e449abda204d6f63ae3
I20260928 14:50:11.925700 32081 ts_tablet_manager.cc:1918] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:11.925850 32081 log.cc:1196] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/357eef3e7ef74e449abda204d6f63ae3
I20260928 14:50:11.926165 32081 ts_tablet_manager.cc:1939] T 357eef3e7ef74e449abda204d6f63ae3 P 11812bed3e8143a69e77be20f362feb4: Deleting consensus metadata
I20260928 14:50:11.926482 32082 ts_tablet_manager.cc:1939] T 357eef3e7ef74e449abda204d6f63ae3 P 15da2ca6043d4e5eb42d0f42a1155c34: Deleting consensus metadata
I20260928 14:50:11.927001 31539 catalog_manager.cc:4841] TS 11812bed3e8143a69e77be20f362feb4 (127.18.96.130:39331): tablet 357eef3e7ef74e449abda204d6f63ae3 (table test-table [id=d043d3df7b554528aba1beedf7ad3226]) successfully deleted
I20260928 14:50:11.927578 31538 catalog_manager.cc:4841] TS 15da2ca6043d4e5eb42d0f42a1155c34 (127.18.96.131:44583): tablet 357eef3e7ef74e449abda204d6f63ae3 (table test-table [id=d043d3df7b554528aba1beedf7ad3226]) successfully deleted
W20260928 14:50:11.933296 31539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:34735 (error 108)
I20260928 14:50:11.934233 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.934111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:34735 (local address 127.0.0.1:35252)
0928 14:50:11.934155 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.934193 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:34735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 14:50:11.934294 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 31757
W20260928 14:50:11.935571 31539 catalog_manager.cc:4584] TS badc2934947c4fcc87d9afae8a509e7c (127.18.96.129:34735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 357eef3e7ef74e449abda204d6f63ae3: Network error: Client connection negotiation failed: client connection to 127.18.96.129:34735: connect: Connection refused (error 111)
I20260928 14:50:11.944128 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 31891
I20260928 14:50:11.955754 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 31520
I20260928 14:50:11.965099 18818 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:50:12.309679 18818 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:50:12.309778 18818 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:50:12.654311 18818 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:50:12.654 UTC [30579] LOG: received smart shutdown request
2026-09-28 14:50:12.658 UTC [30579] LOG: background worker "logical replication launcher" (PID 30588) exited with exit code 1
2026-09-28 14:50:12.658 UTC [30583] LOG: shutting down
2026-09-28 14:50:12.838 UTC [30579] LOG: database system is shut down
2026-09-28T14:50:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49262 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:13 dist-test-slave-j968 krb5kdc[32092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:13 dist-test-slave-j968 krb5kdc[32092](info): set up 2 sockets
Sep 28 14:50:13 dist-test-slave-j968 krb5kdc[32092](info): commencing operation
krb5kdc: starting...
W20260928 14:50:13.997058 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:50:14.118929 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35183
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:35183,127.18.96.189:43695,127.18.96.188:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:14.205637 32105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:14.205853 32105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:14.205875 32105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:14.207041 32105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:14.207086 32105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:14.207100 32105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:14.207113 32105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:14.207162 32105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:14.208849 32105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:35183,127.18.96.189:43695,127.18.96.188:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:14.209115 32105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:14.209352 32105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:14.214329 32110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:14.214335 32111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:14.214352 32105 server_base.cc:1034] running on GCE node
W20260928 14:50:14.214336 32113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:14.214772 32105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:14.215071 32105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:14.216259 32105 hybrid_clock.cc:648] HybridClock initialized: now 1790607014216258 us; error 49 us; skew 500 ppm
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:14.224205 32105 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:50:14.224727 32105 webserver.cc:460] Webserver started at http://127.18.96.190:35451/ using document root <none> and password file <none>
I20260928 14:50:14.224974 32105 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:14.225056 32105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:14.225198 32105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:14.226253 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "50547f76773e49a2bb90b520869ef0da"
format_stamp: "Formatted at 2026-09-28 14:50:14 on dist-test-slave-j968"
server_key: "d197f0e6e6cd8be3ebb19fe7d5ee1a92"
server_key_iv: "2db8a7ca0793e330d81cf41a03946f48"
server_key_version: "encryptionkey@0"
I20260928 14:50:14.226617 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "50547f76773e49a2bb90b520869ef0da"
format_stamp: "Formatted at 2026-09-28 14:50:14 on dist-test-slave-j968"
server_key: "d197f0e6e6cd8be3ebb19fe7d5ee1a92"
server_key_iv: "2db8a7ca0793e330d81cf41a03946f48"
server_key_version: "encryptionkey@0"
I20260928 14:50:14.227985 32105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 14:50:14.228890 32120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:14.229091 32105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:14.229190 32105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "50547f76773e49a2bb90b520869ef0da"
format_stamp: "Formatted at 2026-09-28 14:50:14 on dist-test-slave-j968"
server_key: "d197f0e6e6cd8be3ebb19fe7d5ee1a92"
server_key_iv: "2db8a7ca0793e330d81cf41a03946f48"
server_key_version: "encryptionkey@0"
I20260928 14:50:14.229281 32105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:14.240036 32105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:14.241839 32105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:14.242009 32105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:14.246049 32105 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:35183
I20260928 14:50:14.246080 32172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:35183 every 8 connection(s)
I20260928 14:50:14.246429 32105 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:14.247284 32173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:14.248704 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 32105
I20260928 14:50:14.248824 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:50:14.249042 18818 external_mini_cluster.cc:1428] Setting key fbbddacccce7a1c9c19bb5cdffc430b8
I20260928 14:50:14.248984 32173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.189@KRBTEST.COM" created.
I20260928 14:50:14.259364 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.249692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35183 (local address 127.0.0.1:60244)
0928 14:50:14.249869 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.249913 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:14.250670 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:14.250981 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:14.250983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:14.251225 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:14.251383 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.251387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.252521 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.252522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.253197 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.253199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.253704 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.253705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.253770 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.254302 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:14.254310 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:14.255634 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:14.257774 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.257781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.257783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.257993 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.258301 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.258301 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.258302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.258384 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.258794 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:14.258799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:14.258854 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 14:50:14.258926 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:50:14.259041 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 14:50:14.259377 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.249722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.18.96.190:35183)
0928 14:50:14.249889 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.249898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.250011 (+ 113us) server_negotiation.cc:407] Connection header received
0928 14:50:14.250842 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.250845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.250869 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.250983 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.251473 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.252412 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.253294 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.253605 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.253734 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.255729 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.255736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.255736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.255748 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.257683 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.258155 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.258157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.258158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.258237 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.258487 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.258488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.258488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.258737 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.258802 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.259028 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.259159 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260928 14:50:14.260213 32173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:14.260598 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.260465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:43695 (local address 127.0.0.1:58502)
0928 14:50:14.260517 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.260567 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 14:50:14.260743 32122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
I20260928 14:50:14.261212 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.261114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:43695 (local address 127.0.0.1:58506)
0928 14:50:14.261160 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.261185 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:50:14.261382 32173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.189:43695: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab.
Entry for principal kudu/127.18.96.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab.
I20260928 14:50:14.275381 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.189:43695
--webserver_interface=127.18.96.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab
--principal=kudu/127.18.96.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:35183,127.18.96.189:43695,127.18.96.188:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:14.310642 32173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } attempt: 1
I20260928 14:50:14.311260 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.311081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:43695 (local address 127.0.0.1:58510)
0928 14:50:14.311175 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.311224 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 14:50:14.311708 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.311604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:43695 (local address 127.0.0.1:58516)
0928 14:50:14.311655 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.311684 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:50:14.311825 32173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.189:43695: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
W20260928 14:50:14.361315 32180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:14.361521 32180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:14.361542 32180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:14.362617 32180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:14.362653 32180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:14.362668 32180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:14.362680 32180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:14.362692 32180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:14.364147 32180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:35183,127.18.96.189:43695,127.18.96.188:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab
--principal=kudu/127.18.96.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.189:43695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.96.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:14.364344 32180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:14.364507 32180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:14.370153 32188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:14.370165 32186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:14.370244 32180 server_base.cc:1034] running on GCE node
W20260928 14:50:14.370393 32185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:14.370656 32180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:14.370895 32180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:14.372057 32180 hybrid_clock.cc:648] HybridClock initialized: now 1790607014372041 us; error 25 us; skew 500 ppm
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:14.374568 32180 init.cc:377] Logged in from keytab as kudu/127.18.96.189@KRBTEST.COM (short username kudu)
I20260928 14:50:14.375103 32180 webserver.cc:460] Webserver started at http://127.18.96.189:42773/ using document root <none> and password file <none>
I20260928 14:50:14.375330 32180 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:14.375396 32180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:14.375530 32180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:14.376679 32180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/data/instance:
uuid: "313b0740ad9d4fc59f1f5c04c47eb67a"
format_stamp: "Formatted at 2026-09-28 14:50:14 on dist-test-slave-j968"
server_key: "2cef285625ca589e0a519ce88fd94ebe"
server_key_iv: "02a21371fc8438a20227d08d5a98b2fa"
server_key_version: "encryptionkey@0"
I20260928 14:50:14.377046 32180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/wal/instance:
uuid: "313b0740ad9d4fc59f1f5c04c47eb67a"
format_stamp: "Formatted at 2026-09-28 14:50:14 on dist-test-slave-j968"
server_key: "2cef285625ca589e0a519ce88fd94ebe"
server_key_iv: "02a21371fc8438a20227d08d5a98b2fa"
server_key_version: "encryptionkey@0"
I20260928 14:50:14.378628 32180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:50:14.379470 32195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:14.379693 32180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:14.379803 32180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/wal
uuid: "313b0740ad9d4fc59f1f5c04c47eb67a"
format_stamp: "Formatted at 2026-09-28 14:50:14 on dist-test-slave-j968"
server_key: "2cef285625ca589e0a519ce88fd94ebe"
server_key_iv: "02a21371fc8438a20227d08d5a98b2fa"
server_key_version: "encryptionkey@0"
I20260928 14:50:14.379907 32180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:14.380043 32173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } attempt: 2
I20260928 14:50:14.380486 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.380345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:43695 (local address 127.0.0.1:58520)
0928 14:50:14.380399 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.380457 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:50:14.380825 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.380732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:43695 (local address 127.0.0.1:58532)
0928 14:50:14.380777 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.380802 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 14:50:14.380914 32173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.189:43695: Network error: Client connection negotiation failed: client connection to 127.18.96.189:43695: connect: Connection refused (error 111)
I20260928 14:50:14.390374 32180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:14.392510 32180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:14.392678 32180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:14.397042 32180 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.189:43695
I20260928 14:50:14.397078 32249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.189:43695 every 8 connection(s)
I20260928 14:50:14.397424 32180 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
I20260928 14:50:14.398257 32250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:14.399961 32250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:14.400458 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 32180
I20260928 14:50:14.400630 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-1/wal/instance
I20260928 14:50:14.400868 18818 external_mini_cluster.cc:1428] Setting key 06c5027c0fe072b4207bb6c2a5f36494
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.188@KRBTEST.COM; defaulting to no policy
I20260928 14:50:14.410074 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.400696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.18.96.190:35183)
0928 14:50:14.400758 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.400761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.400979 (+ 218us) server_negotiation.cc:407] Connection header received
0928 14:50:14.401439 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.401441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.401457 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.401548 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.402221 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.402793 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.403971 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.404346 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.404432 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.407022 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.407024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.407024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.407031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.408636 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.409033 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.409034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.409035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.409090 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.409404 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.409405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.409405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.409532 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.409604 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.409887 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.409977 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:50:14.410363 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.400684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35183 (local address 127.0.0.1:60258)
0928 14:50:14.400835 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.400877 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:14.401341 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:14.401601 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:14.401605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:14.401853 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:14.402105 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.402111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.402889 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.402890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.403827 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.403829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.404457 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.404463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.404604 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.405124 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:14.405133 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:14.406894 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:14.408716 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.408724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.408726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.408963 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.409151 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.409151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.409152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.409279 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.409608 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:14.409615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:14.409753 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 14:50:14.409865 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:50:14.410034 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.188@KRBTEST.COM" created.
I20260928 14:50:14.410971 32250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:50:14.419495 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.411294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:43695 (local address 127.0.0.1:58540)
0928 14:50:14.411485 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.411494 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:14.411530 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:14.411696 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:14.411697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:14.411811 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:14.411904 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.411906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.413016 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.413017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.413768 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.413770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.414458 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.414458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.414516 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.414999 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:14.415003 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:14.416519 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:14.418349 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.418350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.418351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.418601 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.418839 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.418840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.418840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.418907 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.419191 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:14.419191 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:14.419243 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:50:14.419323 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:50:14.419419 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260928 14:50:14.419691 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.411357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.18.96.189:43695)
0928 14:50:14.411532 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.411537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.411566 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:50:14.411591 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.411593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.411609 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.411681 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.412028 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.412913 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.413850 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.414363 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.414442 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.416607 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.416615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.416615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.416622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.418258 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.418716 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.418719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.418720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.418776 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.418972 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.418973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.418973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.419118 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.419183 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.419357 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.419514 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260928 14:50:14.420569 32250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:14.420939 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.420829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32842)
0928 14:50:14.420873 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.420919 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 14:50:14.421082 32196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
I20260928 14:50:14.421543 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.421449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32844)
0928 14:50:14.421490 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.421521 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 14:50:14.421643 32250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:40199: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab.
Entry for principal kudu/127.18.96.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab.
I20260928 14:50:14.427176 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.188:40199
--webserver_interface=127.18.96.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab
--principal=kudu/127.18.96.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:35183,127.18.96.189:43695,127.18.96.188:40199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:14.470873 32250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } attempt: 1
I20260928 14:50:14.471444 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.471302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32846)
0928 14:50:14.471365 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.471412 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:50:14.471799 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.471713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32854)
0928 14:50:14.471756 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.471782 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:50:14.471937 32250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:40199: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
I20260928 14:50:14.472065 32173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } attempt: 3
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:50:14.479140 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.472401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.18.96.189:43695)
0928 14:50:14.472440 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.472442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.472454 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:50:14.472554 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.472555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.472568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.472618 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.472961 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.473489 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.474190 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.474415 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.474505 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.476580 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.476581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.476582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.476588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.478007 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.478406 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.478407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.478407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.478478 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.478676 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.478677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.478677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.478782 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.478834 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.479008 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.479078 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:50:14.479182 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.472357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:43695 (local address 127.0.0.1:58552)
0928 14:50:14.472405 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.472416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:14.472471 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:14.472637 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:14.472638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:14.472807 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:14.472881 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.472883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.473572 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.473573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.474109 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.474111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.474506 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.474507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.474589 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.475083 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:14.475088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:14.476500 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:14.478091 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.478092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.478092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.478316 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.478549 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.478550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.478550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.478615 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.478839 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:14.478840 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:14.478892 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:50:14.478973 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:50:14.479077 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 14:50:14.479637 32173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:14.479996 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.479905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32870)
0928 14:50:14.479939 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.479972 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 14:50:14.480306 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.480219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32874)
0928 14:50:14.480261 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.480288 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 14:50:14.480396 32173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:40199: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
I20260928 14:50:14.511536 32173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } attempt: 1
I20260928 14:50:14.512072 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.511920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32880)
0928 14:50:14.511991 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.512037 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 14:50:14.512393 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.512306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32894)
0928 14:50:14.512335 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.512364 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 14:50:14.512494 32173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:40199: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
W20260928 14:50:14.512331 32257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:14.512579 32257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:14.512619 32257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:14.513762 32257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:14.513800 32257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:14.513814 32257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:14.513827 32257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:14.513862 32257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:14.515355 32257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:35183,127.18.96.189:43695,127.18.96.188:40199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab
--principal=kudu/127.18.96.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.188:40199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.96.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:14.515606 32257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:14.515826 32257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:14.520721 32263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:14.520704 32265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:14.520823 32257 server_base.cc:1034] running on GCE node
W20260928 14:50:14.520727 32262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:14.521106 32257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:14.521426 32257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:14.522583 32257 hybrid_clock.cc:648] HybridClock initialized: now 1790607014522572 us; error 29 us; skew 500 ppm
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:14.525188 32257 init.cc:377] Logged in from keytab as kudu/127.18.96.188@KRBTEST.COM (short username kudu)
I20260928 14:50:14.525640 32257 webserver.cc:460] Webserver started at http://127.18.96.188:38037/ using document root <none> and password file <none>
I20260928 14:50:14.525905 32257 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:14.525956 32257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:14.526088 32257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:14.527166 32257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/data/instance:
uuid: "e7bb3accdf0649eaab269d3b532c18d1"
format_stamp: "Formatted at 2026-09-28 14:50:14 on dist-test-slave-j968"
server_key: "1285cc4806455eb19601b822f535d27d"
server_key_iv: "76b9a93a669fdca13adfd6ee445145df"
server_key_version: "encryptionkey@0"
I20260928 14:50:14.527527 32257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/wal/instance:
uuid: "e7bb3accdf0649eaab269d3b532c18d1"
format_stamp: "Formatted at 2026-09-28 14:50:14 on dist-test-slave-j968"
server_key: "1285cc4806455eb19601b822f535d27d"
server_key_iv: "76b9a93a669fdca13adfd6ee445145df"
server_key_version: "encryptionkey@0"
I20260928 14:50:14.528898 32257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:50:14.529697 32272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:14.529893 32257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:14.529979 32257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/wal
uuid: "e7bb3accdf0649eaab269d3b532c18d1"
format_stamp: "Formatted at 2026-09-28 14:50:14 on dist-test-slave-j968"
server_key: "1285cc4806455eb19601b822f535d27d"
server_key_iv: "76b9a93a669fdca13adfd6ee445145df"
server_key_version: "encryptionkey@0"
I20260928 14:50:14.530052 32257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:14.540098 32250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } attempt: 2
I20260928 14:50:14.540596 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.540455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32908)
0928 14:50:14.540520 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.540560 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 14:50:14.541029 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.540857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32922)
0928 14:50:14.540904 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.541005 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 14:50:14.541136 32250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:40199: Network error: Client connection negotiation failed: client connection to 127.18.96.188:40199: connect: Connection refused (error 111)
I20260928 14:50:14.552361 32257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:14.554754 32257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:14.554920 32257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:14.559130 32257 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.188:40199
I20260928 14:50:14.559167 32324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.188:40199 every 8 connection(s)
I20260928 14:50:14.559561 32257 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
I20260928 14:50:14.560699 32325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:14.561915 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 32257
I20260928 14:50:14.562033 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606857360589-18818-0/minicluster-data/master-2/wal/instance
I20260928 14:50:14.562230 18818 external_mini_cluster.cc:1428] Setting key 38afe6622c6f749bbc2b9208df1ff857
I20260928 14:50:14.562501 32325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:50:14.573499 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.563351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.18.96.190:35183)
0928 14:50:14.563423 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.563425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.563646 (+ 221us) server_negotiation.cc:407] Connection header received
0928 14:50:14.563710 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.563711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.563724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.563774 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.564621 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.565363 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.566746 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.567054 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.567123 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.569894 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.569896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.569896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.569906 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.571624 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.572215 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.572217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.572217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.572284 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.572651 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.572652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.572652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.572806 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.572884 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.573293 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.573386 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:50:14.573647 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.563286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35183 (local address 127.0.0.1:60268)
0928 14:50:14.563504 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.563546 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:14.563645 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:14.563821 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:14.563824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:14.564266 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:14.564532 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.564539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.565452 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.565453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.566660 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.566662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.567136 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.567137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.567272 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.567855 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:14.567888 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:14.569731 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:14.571726 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.571733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.571735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.572136 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.572395 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.572396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.572397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.572565 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.572928 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:14.572936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:14.573133 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 14:50:14.573218 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:50:14.573394 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":99,"threads_started":1}
I20260928 14:50:14.574059 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.562747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.18.96.188:40199)
0928 14:50:14.562962 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.562970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.562997 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:50:14.563454 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.563461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.563510 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.563634 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.564481 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.565593 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.566746 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.567218 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.567272 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.570265 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.570273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.570274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.570285 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.572275 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.572780 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.572784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.572784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.572842 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.573089 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.573090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.573090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.573253 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.573397 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.573489 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.573949 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260928 14:50:14.574345 32325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:14.582280 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.574603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:43695 (local address 127.0.0.1:58566)
0928 14:50:14.574653 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.574664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:14.574708 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:14.574894 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:14.574895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:14.575032 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:14.575102 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.575104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.575983 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.575984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.576504 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.576506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.576893 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.576894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.576966 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.577454 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:14.577459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:14.579052 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:14.580985 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.580986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.580986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.581149 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.581371 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.581372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.581372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.581423 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.581951 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:14.581952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:14.581990 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:50:14.582073 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:50:14.582204 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 14:50:14.582336 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.574708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.18.96.189:43695)
0928 14:50:14.574768 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.574771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.574781 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:50:14.574811 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.574812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.574823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.574887 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.575227 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.575913 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.576589 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.576799 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.576891 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.579166 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.579168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.579168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.579176 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.580906 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.581236 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.581237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.581238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.581302 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.581480 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.581480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.581481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.581646 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.581723 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.582124 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.582232 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:50:14.582726 32325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:14.583173 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.575157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.18.96.190:35183)
0928 14:50:14.575195 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.575196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.575205 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:50:14.575229 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.575230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.575240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.575296 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.575772 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.576430 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.577267 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.577466 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.577542 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.579991 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.579992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.579992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.579999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.581653 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.581944 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.581945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.581945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.581995 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.582238 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.582239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.582239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.582377 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.582581 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.582988 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.583090 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:50:14.587621 32173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } attempt: 2
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:50:14.590903 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.583011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32928)
0928 14:50:14.583053 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.583062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:14.583099 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:14.583283 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:14.583284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:14.583428 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:14.583456 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.583457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.584534 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.584535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.585291 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.585293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.585754 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.585755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.585793 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.586190 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:14.586195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:14.587422 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:14.589457 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.589458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.589459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.589930 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.590161 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.590161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.590162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.590233 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.590516 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:14.590517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:14.590565 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:50:14.590639 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:50:14.590743 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 14:50:14.590971 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.583045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.18.96.188:40199)
0928 14:50:14.583082 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.583084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.583093 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:50:14.583152 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.583153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.583163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.583242 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.583563 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.584425 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.585366 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.585680 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.585759 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.587502 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.587504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.587504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.587513 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.589371 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.590021 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.590022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.590022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.590092 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.590307 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.590308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.590308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.590442 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.590514 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.590675 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.590816 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":22}
I20260928 14:50:14.591792 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.583795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.18.96.189:43695)
0928 14:50:14.583829 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.583831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.583839 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:50:14.583913 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.583914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.583923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.583981 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.584539 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.585249 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.586126 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.586328 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.586381 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.588915 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.588917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.588917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.588924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.590547 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.590944 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.590945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.590946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.591003 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.591219 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.591220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.591220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.591369 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.591562 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.591608 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.591684 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 14:50:14.592725 32325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1: Bootstrap starting.
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:50:14.593528 32325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:14.593875 32325 log.cc:824] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:14.595073 32325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1: No bootstrap required, opened a new log
I20260928 14:50:14.596125 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:14.596942 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.587907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32932)
0928 14:50:14.587957 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.587969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:14.588012 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:14.588280 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:14.588281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:14.588451 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:14.588499 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.588501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.589914 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.589915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.590783 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.590786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.591799 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.591800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.591902 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.592295 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:14.592300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:14.593624 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:14.595699 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.595700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.595701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.595946 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.596184 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.596185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.596185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.596256 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.596599 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:14.596600 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:14.596647 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:50:14.596728 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:50:14.596837 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:50:14.596971 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.587925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.18.96.188:40199)
0928 14:50:14.588139 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.588142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.588151 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:50:14.588171 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.588172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.588181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.588264 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.588614 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.589820 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.590878 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.591687 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.591792 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.593712 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.593713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.593713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.593725 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.595619 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.596022 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.596024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.596024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.596114 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.596330 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.596331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.596331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.596493 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.596582 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.596769 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.596849 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":130,"threads_started":1}
I20260928 14:50:14.597122 32325 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } }
I20260928 14:50:14.597322 32325 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:14.597371 32325 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bb3accdf0649eaab269d3b532c18d1, State: Initialized, Role: FOLLOWER
I20260928 14:50:14.597508 32325 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } }
I20260928 14:50:14.597826 32325 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:14.598242 32336 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.598351 32336 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:14.599334 32173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da: Bootstrap starting.
I20260928 14:50:14.599931 32173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:14.600267 32173 log.cc:824] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da: Log is configured to *not* fsync() on all Append() calls
W20260928 14:50:14.600704 32348 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:14.600766 32348 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:14.601208 32173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da: No bootstrap required, opened a new log
I20260928 14:50:14.600953 32257 master_runner.cc:186] Error getting master registration for 127.18.96.190:35183: OK, instance_id { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" instance_seqno: 1790607014244905 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:50:14.603621 32257 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:50:14.603641 32173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } }
I20260928 14:50:14.603788 32173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:14.603827 32173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50547f76773e49a2bb90b520869ef0da, State: Initialized, Role: FOLLOWER
I20260928 14:50:14.604004 32173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } }
I20260928 14:50:14.604354 32173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:14.604573 32353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.604671 32353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:50:14.606396 32365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:14.606459 32365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:14.606922 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.597712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.18.96.189:43695)
0928 14:50:14.597752 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.597753 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.598073 (+ 320us) server_negotiation.cc:407] Connection header received
0928 14:50:14.598148 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.598149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.598162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.598216 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.599148 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.599720 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.601495 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.601795 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.601845 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.603500 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.603501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.603502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.603509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.605280 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.605818 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.605820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.605820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.605888 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.606172 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.606173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.606174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.606314 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.606419 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.606718 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.606811 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:50:14.609870 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.597692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.18.96.188:40199)
0928 14:50:14.597732 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.597734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.600954 (+ 3220us) server_negotiation.cc:407] Connection header received
0928 14:50:14.601051 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.601052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.601064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.601090 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.601812 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.602521 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.603459 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.603783 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.603829 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.606934 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.606936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.606936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.606943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.608614 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.609015 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.609016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.609017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.609072 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.609294 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.609295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.609295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.609421 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.609500 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.609701 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.609786 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:50:14.607704 32105 master_runner.cc:186] Error getting master registration for 127.18.96.189:43695: OK, instance_id { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" instance_seqno: 1790607014395709 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:50:14.611212 32105 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:50:14.615134 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.600310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.18.96.190:35183)
0928 14:50:14.600350 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.600352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.607105 (+ 6753us) server_negotiation.cc:407] Connection header received
0928 14:50:14.607130 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.607131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.607143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.607196 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.608263 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.609038 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.610191 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.610498 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.610579 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.612092 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.612094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.612094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.612102 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.613821 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.614263 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.614264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.614264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.614325 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.614557 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.614558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.614559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.614693 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.614811 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.614960 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.615043 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:50:14.632279 32250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } attempt: 3
Sep 28 14:50:14 dist-test-slave-j968 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:50:14.641444 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.632640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:40199 (local address 127.0.0.1:32948)
0928 14:50:14.632682 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:14.632697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:14.632761 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:14.632944 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:14.632945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:14.633175 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:14.633233 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.633236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.634196 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.634197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.635105 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:14.635108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.635599 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.635600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:14.635668 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.636570 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:14.636576 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:14.638197 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:14.640211 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.640212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.640212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.640445 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.640714 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:14.640715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:14.640715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:14.640783 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:14.641086 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:14.641087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:14.641139 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:50:14.641221 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:50:14.641335 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 14:50:14.641458 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:14.632709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.18.96.188:40199)
0928 14:50:14.632756 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:14.632758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:14.632767 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:50:14.632854 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:14.632855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:14.632865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:14.632941 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:14.633315 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.634115 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.635199 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:14.635514 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:14.635590 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:14.638294 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:14.638295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:14.638296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:14.638304 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:14.640119 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.640527 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.640529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.640529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.640639 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:14.640855 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:14.640856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:14.640857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:14.641010 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:14.641089 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:14.641263 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:50:14.641353 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:50:14.643615 32250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a: Bootstrap starting.
I20260928 14:50:14.644430 32250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:14.644754 32250 log.cc:824] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:14.648646 32250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a: No bootstrap required, opened a new log
I20260928 14:50:14.650708 32250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } }
I20260928 14:50:14.650863 32250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:14.650898 32250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313b0740ad9d4fc59f1f5c04c47eb67a, State: Initialized, Role: FOLLOWER
I20260928 14:50:14.651032 32250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } }
I20260928 14:50:14.651396 32250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:14.651367 32368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.651578 32368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:50:14.653326 32379 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:14.653445 32379 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:14.657001 32180 master_runner.cc:263] No leader master found from master 313b0740ad9d4fc59f1f5c04c47eb67a
I20260928 14:50:14.657065 32180 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 14:50:14.742803 32126 tablet.cc:2369] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:14.765188 32336 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:14.765344 32336 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } }
I20260928 14:50:14.765678 32336 leader_election.cc:288] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50547f76773e49a2bb90b520869ef0da (127.18.96.190:35183), 313b0740ad9d4fc59f1f5c04c47eb67a (127.18.96.189:43695)
I20260928 14:50:14.766099 32223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7bb3accdf0649eaab269d3b532c18d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" is_pre_election: true
I20260928 14:50:14.766322 32223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bb3accdf0649eaab269d3b532c18d1 in term 0.
I20260928 14:50:14.766330 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7bb3accdf0649eaab269d3b532c18d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50547f76773e49a2bb90b520869ef0da" is_pre_election: true
I20260928 14:50:14.766474 32147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bb3accdf0649eaab269d3b532c18d1 in term 0.
I20260928 14:50:14.766713 32275 leader_election.cc:302] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50547f76773e49a2bb90b520869ef0da, e7bb3accdf0649eaab269d3b532c18d1; no voters:
I20260928 14:50:14.766891 32336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:14.766942 32336 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:14.766968 32336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:14.767819 32336 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } }
I20260928 14:50:14.767995 32336 leader_election.cc:288] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [CANDIDATE]: Term 1 election: Requested vote from peers 50547f76773e49a2bb90b520869ef0da (127.18.96.190:35183), 313b0740ad9d4fc59f1f5c04c47eb67a (127.18.96.189:43695)
I20260928 14:50:14.768167 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7bb3accdf0649eaab269d3b532c18d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50547f76773e49a2bb90b520869ef0da"
I20260928 14:50:14.768224 32223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7bb3accdf0649eaab269d3b532c18d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a"
I20260928 14:50:14.768251 32147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:14.768293 32223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:14.769176 32223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bb3accdf0649eaab269d3b532c18d1 in term 1.
I20260928 14:50:14.769176 32147 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bb3accdf0649eaab269d3b532c18d1 in term 1.
I20260928 14:50:14.769341 32275 leader_election.cc:302] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50547f76773e49a2bb90b520869ef0da, e7bb3accdf0649eaab269d3b532c18d1; no voters:
I20260928 14:50:14.769436 32336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:14.769603 32336 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [term 1 LEADER]: Becoming Leader. State: Replica: e7bb3accdf0649eaab269d3b532c18d1, State: Running, Role: LEADER
I20260928 14:50:14.769706 32336 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } }
I20260928 14:50:14.770126 32336 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7bb3accdf0649eaab269d3b532c18d1. Latest consensus state: current_term: 1 leader_uuid: "e7bb3accdf0649eaab269d3b532c18d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.770200 32336 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:14.770876 32383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:14.771073 32383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:14.772379 32223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [term 1 FOLLOWER]: Refusing update from remote peer e7bb3accdf0649eaab269d3b532c18d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:14.772397 32147 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [term 1 FOLLOWER]: Refusing update from remote peer e7bb3accdf0649eaab269d3b532c18d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:14.772679 32381 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:14.772765 32336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:14.773921 32353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7bb3accdf0649eaab269d3b532c18d1. Latest consensus state: current_term: 1 leader_uuid: "e7bb3accdf0649eaab269d3b532c18d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.773921 32368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7bb3accdf0649eaab269d3b532c18d1. Latest consensus state: current_term: 1 leader_uuid: "e7bb3accdf0649eaab269d3b532c18d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.774003 32353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:14.774012 32368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:14.774883 32336 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7bb3accdf0649eaab269d3b532c18d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.774968 32336 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:14.775357 32353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7bb3accdf0649eaab269d3b532c18d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.775440 32353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:14.776079 32336 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7bb3accdf0649eaab269d3b532c18d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.776156 32336 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:14.776671 32368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7bb3accdf0649eaab269d3b532c18d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50547f76773e49a2bb90b520869ef0da" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35183 } } peers { permanent_uuid: "313b0740ad9d4fc59f1f5c04c47eb67a" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 43695 } } peers { permanent_uuid: "e7bb3accdf0649eaab269d3b532c18d1" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 40199 } } }
I20260928 14:50:14.776764 32368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:14.776921 32386 mvcc.cc:205] Tried to move back new op lower bound from 7334326332505706496 to 7334326332497436672. Current Snapshot: MvccSnapshot[applied={T|T < 7334326332505706496}]
I20260928 14:50:14.777088 32383 catalog_manager.cc:1347] Generated new cluster ID: 82f76bcf45504cd6a2ee26dc307bd58b
I20260928 14:50:14.777146 32383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:14.789971 32383 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:14.790627 32383 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:14.800927 32383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7bb3accdf0649eaab269d3b532c18d1: Generated new TSK 0
I20260928 14:50:14.801129 32383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:15.606889 32365 catalog_manager.cc:1259] Loaded cluster ID: 82f76bcf45504cd6a2ee26dc307bd58b
I20260928 14:50:15.606971 32365 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da: loading cluster ID for follower catalog manager: success
I20260928 14:50:15.607846 32365 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da: acquiring CA information for follower catalog manager: success
I20260928 14:50:15.608223 32365 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50547f76773e49a2bb90b520869ef0da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:50:15.653967 32379 catalog_manager.cc:1259] Loaded cluster ID: 82f76bcf45504cd6a2ee26dc307bd58b
I20260928 14:50:15.654101 32379 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a: loading cluster ID for follower catalog manager: success
I20260928 14:50:15.655066 32379 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a: acquiring CA information for follower catalog manager: success
I20260928 14:50:15.655416 32379 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 313b0740ad9d4fc59f1f5c04c47eb67a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:50:15.855610 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 32105
I20260928 14:50:15.861151 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 32180
I20260928 14:50:15.866318 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 32257
2026-09-28T14:50:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2910 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:15 dist-test-slave-j968 krb5kdc[32399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:15 dist-test-slave-j968 krb5kdc[32399](info): set up 2 sockets
Sep 28 14:50:15 dist-test-slave-j968 krb5kdc[32399](info): commencing operation
krb5kdc: starting...
W20260928 14:50:16.916946 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:50:17 dist-test-slave-j968 krb5kdc[32399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:50:17.042559 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:41101
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:17.124591 32412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:17.124809 32412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:17.124832 32412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:17.125921 32412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:17.125959 32412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:17.125973 32412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:17.125988 32412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:17.125999 32412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:17.127401 32412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:41101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:17.127604 32412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:17.127786 32412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:17.132655 32420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:17.132670 32418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:17.132668 32417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:17.132761 32412 server_base.cc:1034] running on GCE node
I20260928 14:50:17.133138 32412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:17.133435 32412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:17.134584 32412 hybrid_clock.cc:648] HybridClock initialized: now 1790607017134562 us; error 44 us; skew 500 ppm
Sep 28 14:50:17 dist-test-slave-j968 krb5kdc[32399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607017, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:17.137323 32412 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:50:17.137825 32412 webserver.cc:460] Webserver started at http://127.18.96.190:34235/ using document root <none> and password file <none>
I20260928 14:50:17.138060 32412 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:17.138106 32412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:17.138279 32412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:17.139439 32412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "2d8f7a7032a047f897f29b7191891fcb"
format_stamp: "Formatted at 2026-09-28 14:50:17 on dist-test-slave-j968"
server_key: "d10477af43338be118cdc0ee5ec8535b"
server_key_iv: "d73b09f5430d1132a00c67b0a8402549"
server_key_version: "encryptionkey@0"
I20260928 14:50:17.139850 32412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "2d8f7a7032a047f897f29b7191891fcb"
format_stamp: "Formatted at 2026-09-28 14:50:17 on dist-test-slave-j968"
server_key: "d10477af43338be118cdc0ee5ec8535b"
server_key_iv: "d73b09f5430d1132a00c67b0a8402549"
server_key_version: "encryptionkey@0"
I20260928 14:50:17.141232 32412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:17.142114 32427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:17.142313 32412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:17.142413 32412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "2d8f7a7032a047f897f29b7191891fcb"
format_stamp: "Formatted at 2026-09-28 14:50:17 on dist-test-slave-j968"
server_key: "d10477af43338be118cdc0ee5ec8535b"
server_key_iv: "d73b09f5430d1132a00c67b0a8402549"
server_key_version: "encryptionkey@0"
I20260928 14:50:17.142483 32412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:17.153529 32412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:17.155398 32412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:17.155575 32412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:17.159852 32412 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:41101
I20260928 14:50:17.159936 32479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:41101 every 8 connection(s)
I20260928 14:50:17.160331 32412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:17.161180 32480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:17.163421 32480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb: Bootstrap starting.
I20260928 14:50:17.164037 32480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:17.164273 32480 log.cc:824] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:17.164921 32480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb: No bootstrap required, opened a new log
I20260928 14:50:17.166432 32480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8f7a7032a047f897f29b7191891fcb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41101 } }
I20260928 14:50:17.166568 32480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:17.166590 32480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8f7a7032a047f897f29b7191891fcb, State: Initialized, Role: FOLLOWER
I20260928 14:50:17.166738 32480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8f7a7032a047f897f29b7191891fcb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41101 } }
I20260928 14:50:17.166828 32480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:17.166852 32480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:17.166900 32480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:17.167610 32480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8f7a7032a047f897f29b7191891fcb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41101 } }
I20260928 14:50:17.167646 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 32412
I20260928 14:50:17.167760 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:50:17.167747 32480 leader_election.cc:302] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d8f7a7032a047f897f29b7191891fcb; no voters:
I20260928 14:50:17.167925 32480 leader_election.cc:288] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:17.168030 18818 external_mini_cluster.cc:1428] Setting key fb2e5d856919a1cb32e7eac474e27971
I20260928 14:50:17.168215 32484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:17.168234 32480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:17.168447 32484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [term 1 LEADER]: Becoming Leader. State: Replica: 2d8f7a7032a047f897f29b7191891fcb, State: Running, Role: LEADER
I20260928 14:50:17.168612 32484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8f7a7032a047f897f29b7191891fcb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41101 } }
I20260928 14:50:17.169086 32483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d8f7a7032a047f897f29b7191891fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8f7a7032a047f897f29b7191891fcb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41101 } } }
I20260928 14:50:17.169173 32483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:17.169073 32484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d8f7a7032a047f897f29b7191891fcb. Latest consensus state: current_term: 1 leader_uuid: "2d8f7a7032a047f897f29b7191891fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d8f7a7032a047f897f29b7191891fcb" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41101 } } }
I20260928 14:50:17.169399 32494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:17.169471 32484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:17.170205 32494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:17.171797 32494 catalog_manager.cc:1347] Generated new cluster ID: 337a21abda414ef081019ac3a6a5db8b
I20260928 14:50:17.171849 32494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:50:17 dist-test-slave-j968 krb5kdc[32399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:17.180195 32487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.168668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.18.96.190:41101)
0928 14:50:17.169824 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:17.169834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:17.169877 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:50:17.170001 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:17.170009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:17.170065 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:17.170202 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:17.171349 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.172462 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.173248 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.173709 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.173841 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.176279 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:17.176283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:17.176283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:17.176295 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:17.178517 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:17.178902 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:17.178905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:17.178906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:17.179016 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:17.179230 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:17.179231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:17.179231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:17.179502 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:17.179694 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:17.179797 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 14:50:17.179978 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103,"thread_start_us":68,"threads_started":1}
I20260928 14:50:17.181169 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:17.186508 32494 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:17.187021 32494 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:17.191672 32494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d8f7a7032a047f897f29b7191891fcb: Generated new TSK 0
I20260928 14:50:17.191891 32494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:17.235373 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 32412
2026-09-28T14:50:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1367 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:17 dist-test-slave-j968 krb5kdc[32508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:17 dist-test-slave-j968 krb5kdc[32508](info): set up 2 sockets
Sep 28 14:50:17 dist-test-slave-j968 krb5kdc[32508](info): commencing operation
krb5kdc: starting...
W20260928 14:50:18.265434 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:50:18 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:50:18.391248 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43163
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:43163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:18.477897 32521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:18.478156 32521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:18.478195 32521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:18.479326 32521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:18.479396 32521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:18.479423 32521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:18.479454 32521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:18.479477 32521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:18.481010 32521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:43163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:18.481276 32521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:18.481496 32521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:18.486481 32529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:18.486522 32521 server_base.cc:1034] running on GCE node
W20260928 14:50:18.486460 32526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:18.486460 32527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:18.486943 32521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:18.487248 32521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:18.488416 32521 hybrid_clock.cc:648] HybridClock initialized: now 1790607018488400 us; error 45 us; skew 500 ppm
Sep 28 14:50:18 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:18.490950 32521 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:50:18.491412 32521 webserver.cc:460] Webserver started at http://127.18.96.190:33365/ using document root <none> and password file <none>
I20260928 14:50:18.491640 32521 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:18.491726 32521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:18.491937 32521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:18.493016 32521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "75074b08af9943de801717c7f5c975e4"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "b829726da2a5547fe13a67606c27bbb4"
server_key_iv: "a1789a6e9b9a443a9882f25861d037e1"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.493382 32521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "75074b08af9943de801717c7f5c975e4"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "b829726da2a5547fe13a67606c27bbb4"
server_key_iv: "a1789a6e9b9a443a9882f25861d037e1"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.494714 32521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:50:18.495478 32536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:18.495653 32521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:18.495757 32521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "75074b08af9943de801717c7f5c975e4"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "b829726da2a5547fe13a67606c27bbb4"
server_key_iv: "a1789a6e9b9a443a9882f25861d037e1"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.495839 32521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:18.509089 32521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:18.510982 32521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:18.511152 32521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:18.515252 32521 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:43163
I20260928 14:50:18.515301 32588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:43163 every 8 connection(s)
I20260928 14:50:18.515668 32521 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:18.516534 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 32521
I20260928 14:50:18.516624 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:50:18.516830 32589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:18.516876 18818 external_mini_cluster.cc:1428] Setting key 92035847888f7e55cb104d4a460d919e
I20260928 14:50:18.520411 32589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Bootstrap starting.
I20260928 14:50:18.521045 32589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:18.521351 32589 log.cc:824] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:18.522183 32589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: No bootstrap required, opened a new log
Sep 28 14:50:18 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:18.524138 32589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:18.524326 32589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:18.524367 32589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75074b08af9943de801717c7f5c975e4, State: Initialized, Role: FOLLOWER
I20260928 14:50:18.524511 32589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:18.524631 32589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:18.524683 32589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:18.524734 32589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:18.525481 32589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:18.525595 32589 leader_election.cc:302] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75074b08af9943de801717c7f5c975e4; no voters:
I20260928 14:50:18.525756 32589 leader_election.cc:288] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:18.525815 32594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:18.525981 32594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 1 LEADER]: Becoming Leader. State: Replica: 75074b08af9943de801717c7f5c975e4, State: Running, Role: LEADER
I20260928 14:50:18.526130 32594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:18.526019 32589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:18.527498 32597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75074b08af9943de801717c7f5c975e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } } }
I20260928 14:50:18.527593 32597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:18.528090 32595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75074b08af9943de801717c7f5c975e4. Latest consensus state: current_term: 1 leader_uuid: "75074b08af9943de801717c7f5c975e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } } }
I20260928 14:50:18.528187 32595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:18.528272 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:18.517413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.18.96.190:43163)
0928 14:50:18.517597 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:18.517609 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:18.517642 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:50:18.518091 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:18.518100 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:18.518163 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:18.518319 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:18.518896 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.520298 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.521063 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.521669 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.521751 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:18.524172 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:18.524176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:18.524176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:18.524188 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:18.526289 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.526700 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.526703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.526704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.526819 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.527017 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.527018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.527018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.527293 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:18.527615 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:18.527764 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 14:50:18.527964 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
W20260928 14:50:18.528316 32611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:18.528350 32611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:18.528396 32608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:18.528530 32608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:18.529953 32608 catalog_manager.cc:1347] Generated new cluster ID: 8cb3e024d27b4fdd8497975ae47ec906
I20260928 14:50:18.530009 32608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:18.539175 32608 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:18.539949 32608 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:18.548846 32608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Generated new TSK 0
I20260928 14:50:18.549057 32608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:50:18.580096 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43163
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:18.664268 32616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:18.664563 32616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:18.664593 32616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:18.665794 32616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:18.665836 32616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:18.665863 32616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:50:18.667292 32616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:18.667498 32616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:18.667685 32616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:18.672829 32624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:18.672840 32622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:18.672871 32621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:18.673043 32616 server_base.cc:1034] running on GCE node
I20260928 14:50:18.673287 32616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:18.673548 32616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:18.674722 32616 hybrid_clock.cc:648] HybridClock initialized: now 1790607018674703 us; error 30 us; skew 500 ppm
Sep 28 14:50:18 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:18.677582 32616 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:50:18.678077 32616 webserver.cc:460] Webserver started at http://127.18.96.129:35751/ using document root <none> and password file <none>
I20260928 14:50:18.678337 32616 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:18.678408 32616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:18.678591 32616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:18.679693 32616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "d3400aa1331c42da855345c407bf5eba"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "02e8ae15200cf517ef6fbad5e02036dd"
server_key_iv: "fcaa3f47b673c1eefdb63759a45f9dcb"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.680091 32616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "d3400aa1331c42da855345c407bf5eba"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "02e8ae15200cf517ef6fbad5e02036dd"
server_key_iv: "fcaa3f47b673c1eefdb63759a45f9dcb"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.681525 32616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:18.682379 32631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:18.682569 32616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:18.682684 32616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "d3400aa1331c42da855345c407bf5eba"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "02e8ae15200cf517ef6fbad5e02036dd"
server_key_iv: "fcaa3f47b673c1eefdb63759a45f9dcb"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.682766 32616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:18.696290 32616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:18.698552 32616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:18.698676 32616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:18.698868 32616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:18.699189 32616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:18.699223 32616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:18.699275 32616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:18.699301 32616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:18.705421 32616 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:36059
I20260928 14:50:18.705440 32745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:36059 every 8 connection(s)
I20260928 14:50:18.705889 32616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:50:18 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:18.715315 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 32616
I20260928 14:50:18.715462 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:50:18.715692 18818 external_mini_cluster.cc:1428] Setting key 28c2843f0a26df3dc54590ffca0a1cf7
I20260928 14:50:18.716120 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:18.706770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:55541 (local address 127.18.96.190:43163)
0928 14:50:18.706822 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:18.706823 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:18.707088 (+ 265us) server_negotiation.cc:407] Connection header received
0928 14:50:18.707629 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:18.707631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:18.707642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:18.707707 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:18.708796 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.709397 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.710535 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.710810 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.710863 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:18.713199 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:18.713201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:18.713201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:18.713209 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:18.714517 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.714957 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.714958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.714958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.715010 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.715304 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.715306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.715306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.715459 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:18.715563 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:18.715912 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 14:50:18.716014 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:50:18.716379 32748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:18.706729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:55541)
0928 14:50:18.706914 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:18.706963 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:18.707531 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:18.707787 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:18.707790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:18.708476 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:18.708688 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:18.708694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.709519 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.709520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:18.710425 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:18.710429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.710886 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.710886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:18.710998 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:18.711517 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:18.711525 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:18.713077 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:18.714610 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:18.714618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:18.714620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:18.714884 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:18.715072 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:18.715073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:18.715073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:18.715192 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:18.715561 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:18.715569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:18.715718 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 14:50:18.715857 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 14:50:18.716133 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260928 14:50:18.716856 32746 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43163
I20260928 14:50:18.716985 32746 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:18.717181 32746 heartbeater.cc:511] Master 127.18.96.190:43163 requested a full tablet report, sending...
I20260928 14:50:18.717756 32553 ts_manager.cc:194] Registered new tserver with Master: d3400aa1331c42da855345c407bf5eba (127.18.96.129:36059)
I20260928 14:50:18.718729 32553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:55541
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:50:18.743249 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43163
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:18.825975 32751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:18.826201 32751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:18.826227 32751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:18.827540 32751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:18.827584 32751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:18.827611 32751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:50:18.829115 32751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:18.829319 32751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:18.829500 32751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:18.834930 32759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:18.834930 32757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:18.835206 32751 server_base.cc:1034] running on GCE node
W20260928 14:50:18.835212 32756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:18.835500 32751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:18.835767 32751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:18.836934 32751 hybrid_clock.cc:648] HybridClock initialized: now 1790607018836940 us; error 50 us; skew 500 ppm
Sep 28 14:50:18 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:18.839915 32751 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:50:18.840479 32751 webserver.cc:460] Webserver started at http://127.18.96.130:46377/ using document root <none> and password file <none>
I20260928 14:50:18.840824 32751 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:18.840895 32751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:18.841079 32751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:18.842710 32751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "5fa991f12e5943eeb335769d1e36c2db"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "17c3c08b22f950f0a0a19240fd1c7cdc"
server_key_iv: "a0dfce3c26b0dc75d42b9eee14c019cb"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.843092 32751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "5fa991f12e5943eeb335769d1e36c2db"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "17c3c08b22f950f0a0a19240fd1c7cdc"
server_key_iv: "a0dfce3c26b0dc75d42b9eee14c019cb"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.844530 32751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 14:50:18.845263 32766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:18.845498 32751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:18.845574 32751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "5fa991f12e5943eeb335769d1e36c2db"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "17c3c08b22f950f0a0a19240fd1c7cdc"
server_key_iv: "a0dfce3c26b0dc75d42b9eee14c019cb"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.845629 32751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:18.864291 32751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:18.866291 32751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:18.866408 32751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:18.866613 32751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:18.866865 32751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:18.866894 32751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:18.866918 32751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:18.866931 32751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:18.872964 32751 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:33115
I20260928 14:50:18.873046 413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:33115 every 8 connection(s)
I20260928 14:50:18.873382 32751 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:18.878650 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 32751
I20260928 14:50:18.878780 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:50:18.879000 18818 external_mini_cluster.cc:1428] Setting key 3de9eaa108d37ada8a8bb86ad73656f6
Sep 28 14:50:18 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:18.884145 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:18.874246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:42199 (local address 127.18.96.190:43163)
0928 14:50:18.874413 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:18.874418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:18.874604 (+ 186us) server_negotiation.cc:407] Connection header received
0928 14:50:18.875209 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:18.875210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:18.875221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:18.875278 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:18.876280 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.876777 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.877806 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.878016 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.878074 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:18.880827 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:18.880828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:18.880829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:18.880835 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:18.882607 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.883025 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.883026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.883027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.883086 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.883379 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.883380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.883381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.883543 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:18.883615 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:18.883898 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 14:50:18.883996 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260928 14:50:18.884251 416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:18.874149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:42199)
0928 14:50:18.874492 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:18.874530 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:18.875097 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:18.875333 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:18.875336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:18.875921 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:18.876198 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:18.876203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.876863 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.876864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:18.877724 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:18.877726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.878099 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.878100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:18.878204 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:18.878804 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:18.878814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:18.880736 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:18.882705 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:18.882712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:18.882714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:18.882956 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:18.883175 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:18.883176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:18.883176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:18.883307 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:18.883612 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:18.883616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:18.883743 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 14:50:18.883827 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:50:18.884046 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":194,"threads_started":1}
I20260928 14:50:18.884740 414 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43163
I20260928 14:50:18.884846 414 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:18.885061 414 heartbeater.cc:511] Master 127.18.96.190:43163 requested a full tablet report, sending...
I20260928 14:50:18.885447 32553 ts_manager.cc:194] Registered new tserver with Master: 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115)
I20260928 14:50:18.885859 32553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:42199
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:50:18.907112 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43163
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:18.989616 419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:18.989871 419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:18.989914 419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:18.991078 419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:18.991150 419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:18.991205 419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:50:18.992692 419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:43163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:18.992978 419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:18.993232 419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:18.998605 425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:18.998620 427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:18.998698 419 server_base.cc:1034] running on GCE node
W20260928 14:50:18.998624 424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:18.998984 419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:18.999253 419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:19.000381 419 hybrid_clock.cc:648] HybridClock initialized: now 1790607019000367 us; error 34 us; skew 500 ppm
Sep 28 14:50:18 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:19.003021 419 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:50:19.003507 419 webserver.cc:460] Webserver started at http://127.18.96.131:36341/ using document root <none> and password file <none>
I20260928 14:50:19.003764 419 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:19.003811 419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:19.003965 419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:19.005061 419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "7997523c12664e30a31e70902792f240"
format_stamp: "Formatted at 2026-09-28 14:50:19 on dist-test-slave-j968"
server_key: "cfe741fcb5059e4be27d6732558227c9"
server_key_iv: "0fc88a765341e6f7f5642c4f73ac95a7"
server_key_version: "encryptionkey@0"
I20260928 14:50:19.005448 419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "7997523c12664e30a31e70902792f240"
format_stamp: "Formatted at 2026-09-28 14:50:19 on dist-test-slave-j968"
server_key: "cfe741fcb5059e4be27d6732558227c9"
server_key_iv: "0fc88a765341e6f7f5642c4f73ac95a7"
server_key_version: "encryptionkey@0"
I20260928 14:50:19.006814 419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:50:19.007671 434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.007942 419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:19.008045 419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "7997523c12664e30a31e70902792f240"
format_stamp: "Formatted at 2026-09-28 14:50:19 on dist-test-slave-j968"
server_key: "cfe741fcb5059e4be27d6732558227c9"
server_key_iv: "0fc88a765341e6f7f5642c4f73ac95a7"
server_key_version: "encryptionkey@0"
I20260928 14:50:19.008131 419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:19.015440 419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:19.017771 419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:19.017935 419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:19.018175 419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:19.018491 419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:19.018544 419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.018586 419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:19.018617 419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.024639 419 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:46743
I20260928 14:50:19.024726 548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:46743 every 8 connection(s)
I20260928 14:50:19.025061 419 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:50:19 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:19.032300 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 419
I20260928 14:50:19.032397 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:50:19.032680 18818 external_mini_cluster.cc:1428] Setting key e5cd6bd69f2fb461c8574d187fa80de3
I20260928 14:50:19.035583 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.025827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:43499 (local address 127.18.96.190:43163)
0928 14:50:19.025893 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.025895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.026134 (+ 239us) server_negotiation.cc:407] Connection header received
0928 14:50:19.026659 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.026661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.026672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.026734 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:19.028051 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.028730 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.029789 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.030024 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.030096 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.032632 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:19.032634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:19.032634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:19.032641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:19.034337 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.034744 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.034744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.034745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.034796 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.035040 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.035040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.035040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.035139 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:19.035198 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.035446 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.035519 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:50:19.035789 551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.025818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:43499)
0928 14:50:19.025975 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.026025 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.026551 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.026834 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.026838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.027637 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:19.027938 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.027946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.028829 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.028830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.029695 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.029697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.030146 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.030147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.030261 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.030757 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:19.030766 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:19.032533 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:19.034423 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.034430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.034431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.034673 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.034839 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.034839 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.034840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.034927 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.035199 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:19.035205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:19.035329 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.035420 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.035554 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260928 14:50:19.036294 549 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43163
I20260928 14:50:19.036379 549 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:19.036578 549 heartbeater.cc:511] Master 127.18.96.190:43163 requested a full tablet report, sending...
I20260928 14:50:19.037005 32553 ts_manager.cc:194] Registered new tserver with Master: 7997523c12664e30a31e70902792f240 (127.18.96.131:46743)
I20260928 14:50:19.037379 32553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:43499
I20260928 14:50:19.044683 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:19.053047 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.046241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.18.96.190:43163)
0928 14:50:19.046325 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.046327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.046438 (+ 111us) server_negotiation.cc:407] Connection header received
0928 14:50:19.046483 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.046484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.046500 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.046557 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:19.047163 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.047734 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.048600 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.048820 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.048938 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.050466 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:19.050467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:19.050468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:19.050475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:19.051779 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.052224 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.052225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.052226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.052280 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.052464 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.052465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.052465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.052609 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:19.052684 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.052788 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.052853 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:50:19.054914 32553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:19.056748 32553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:19.063752 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.059399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:33115 (local address 127.0.0.1:36462)
0928 14:50:19.059528 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.059570 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.059633 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.060117 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.060121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.060123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:19.060196 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.060198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.061531 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.061532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.062626 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.062628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.063431 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.063432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.063512 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.063526 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.063604 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.063658 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":82,"threads_started":1}
I20260928 14:50:19.063773 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.059533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.18.96.130:33115)
0928 14:50:19.059759 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.059765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.059776 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:50:19.059823 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.059825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.059842 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.060015 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:19.060261 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.061410 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.062717 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.063341 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.063423 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.063481 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.063635 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.063687 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":112,"threads_started":1}
I20260928 14:50:19.064189 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.059635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:46743 (local address 127.0.0.1:57060)
0928 14:50:19.060065 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.060075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.060132 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.060410 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.060411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.060413 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:19.060498 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.060500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.061771 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.061772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.063023 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.063025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.063923 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.063924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.064001 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.064005 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.064083 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.064123 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":36,"thread_start_us":74,"threads_started":1}
I20260928 14:50:19.064219 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.059751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.18.96.131:46743)
0928 14:50:19.060267 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.060272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.060280 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:50:19.060310 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.060312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.060324 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.060412 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:19.060603 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.061642 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.063134 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.063770 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.063835 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.063917 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.064102 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.064142 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":75,"threads_started":1}
I20260928 14:50:19.064641 346 tablet_service.cc:1467] Processing CreateTablet for tablet 7796068140f64e05a9b6630b26afa131 (DEFAULT_TABLE table=test-table [id=7537b1e7a6394af4a6642fae25a2a936]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:19.064800 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.059832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36059 (local address 127.0.0.1:39950)
0928 14:50:19.060303 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.060314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.060353 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.060736 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.060737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.060739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:19.060776 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.060777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.061724 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.061725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.063021 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.063025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.064598 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.064599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.064656 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.064660 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.064719 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.064751 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":66,"thread_start_us":96,"threads_started":1}
I20260928 14:50:19.064869 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.059983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.18.96.129:36059)
0928 14:50:19.060560 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.060565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.060574 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:50:19.060613 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.060615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.060630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.060725 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:19.060831 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.061605 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.063115 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.064502 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.064577 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.064630 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.064745 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.064785 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":72,"threads_started":1}
I20260928 14:50:19.065053 346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7796068140f64e05a9b6630b26afa131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:19.067655 567 tablet_bootstrap.cc:492] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: Bootstrap starting.
I20260928 14:50:19.068325 483 tablet_service.cc:1467] Processing CreateTablet for tablet 7796068140f64e05a9b6630b26afa131 (DEFAULT_TABLE table=test-table [id=7537b1e7a6394af4a6642fae25a2a936]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:19.068332 32679 tablet_service.cc:1467] Processing CreateTablet for tablet 7796068140f64e05a9b6630b26afa131 (DEFAULT_TABLE table=test-table [id=7537b1e7a6394af4a6642fae25a2a936]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:19.068626 483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7796068140f64e05a9b6630b26afa131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:19.068643 32679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7796068140f64e05a9b6630b26afa131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:19.070992 568 tablet_bootstrap.cc:492] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: Bootstrap starting.
I20260928 14:50:19.071446 569 tablet_bootstrap.cc:492] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Bootstrap starting.
I20260928 14:50:19.071688 567 tablet_bootstrap.cc:654] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:19.071774 568 tablet_bootstrap.cc:654] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:19.072139 569 tablet_bootstrap.cc:654] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:19.072256 567 log.cc:824] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:19.072358 568 log.cc:824] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:19.072706 569 log.cc:824] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:19.073078 567 tablet_bootstrap.cc:492] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: No bootstrap required, opened a new log
I20260928 14:50:19.073184 567 ts_tablet_manager.cc:1397] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260928 14:50:19.073447 569 tablet_bootstrap.cc:492] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: No bootstrap required, opened a new log
I20260928 14:50:19.073484 568 tablet_bootstrap.cc:492] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: No bootstrap required, opened a new log
I20260928 14:50:19.073515 569 ts_tablet_manager.cc:1397] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:50:19.073575 568 ts_tablet_manager.cc:1397] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:50:19.075129 568 raft_consensus.cc:348] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.075286 568 raft_consensus.cc:374] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:19.075330 568 raft_consensus.cc:729] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7997523c12664e30a31e70902792f240, State: Initialized, Role: FOLLOWER
I20260928 14:50:19.075313 567 raft_consensus.cc:348] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.075489 567 raft_consensus.cc:374] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:19.075451 568 consensus_queue.cc:260] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.075531 567 raft_consensus.cc:729] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa991f12e5943eeb335769d1e36c2db, State: Initialized, Role: FOLLOWER
I20260928 14:50:19.075431 569 raft_consensus.cc:348] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.075577 569 raft_consensus.cc:374] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:19.075608 569 raft_consensus.cc:729] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3400aa1331c42da855345c407bf5eba, State: Initialized, Role: FOLLOWER
I20260928 14:50:19.075649 567 consensus_queue.cc:260] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.075784 568 ts_tablet_manager.cc:1428] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:50:19.075749 569 consensus_queue.cc:260] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.075966 549 heartbeater.cc:503] Master 127.18.96.190:43163 was elected leader, sending a full tablet report...
I20260928 14:50:19.075982 567 ts_tablet_manager.cc:1428] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:50:19.076032 569 ts_tablet_manager.cc:1428] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 14:50:19.076113 414 heartbeater.cc:503] Master 127.18.96.190:43163 was elected leader, sending a full tablet report...
I20260928 14:50:19.076143 32746 heartbeater.cc:503] Master 127.18.96.190:43163 was elected leader, sending a full tablet report...
W20260928 14:50:19.123986 415 tablet.cc:2369] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:50:19.206801 32747 tablet.cc:2369] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:19.260594 575 raft_consensus.cc:482] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:19.260725 575 raft_consensus.cc:504] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.261138 575 leader_election.cc:288] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115), 7997523c12664e30a31e70902792f240 (127.18.96.131:46743)
I20260928 14:50:19.264410 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.261436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:36687 (local address 127.18.96.130:33115)
0928 14:50:19.261624 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.261627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.261720 (+ 93us) server_negotiation.cc:407] Connection header received
0928 14:50:19.261795 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.261797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.261813 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.261863 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:19.262104 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.262582 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.263600 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.264090 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.264146 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.264153 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.264323 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.264357 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260928 14:50:19.264438 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.261350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:33115 (local address 127.18.96.129:36687)
0928 14:50:19.261619 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.261633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.261726 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.261887 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.261888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.261890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:19.262006 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.262010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.262661 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.262662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.263497 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.263499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.264167 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.264169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.264242 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.264246 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.264307 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.264348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":91,"threads_started":1}
I20260928 14:50:19.264854 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.261572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:46743 (local address 127.18.96.129:41885)
0928 14:50:19.261679 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.261690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.261742 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.261920 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.261921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.261923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:19.262022 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.262025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.262839 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.262840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.263749 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.263752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.264618 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.264619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.264683 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.264686 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.264737 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.264770 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260928 14:50:19.264868 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.261663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:41885 (local address 127.18.96.131:46743)
0928 14:50:19.261734 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.261736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.261747 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:50:19.261801 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.261802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.261817 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.261895 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:19.262114 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.262722 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.263839 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.264504 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.264576 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.264585 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.264762 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.264803 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:50:19.264811 366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7796068140f64e05a9b6630b26afa131" candidate_uuid: "d3400aa1331c42da855345c407bf5eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa991f12e5943eeb335769d1e36c2db" is_pre_election: true
I20260928 14:50:19.265064 366 raft_consensus.cc:2454] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3400aa1331c42da855345c407bf5eba in term 0.
I20260928 14:50:19.265295 503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7796068140f64e05a9b6630b26afa131" candidate_uuid: "d3400aa1331c42da855345c407bf5eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7997523c12664e30a31e70902792f240" is_pre_election: true
I20260928 14:50:19.265364 32635 leader_election.cc:302] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa991f12e5943eeb335769d1e36c2db, d3400aa1331c42da855345c407bf5eba; no voters:
I20260928 14:50:19.265476 503 raft_consensus.cc:2454] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3400aa1331c42da855345c407bf5eba in term 0.
I20260928 14:50:19.265530 575 raft_consensus.cc:2781] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:19.265622 575 raft_consensus.cc:482] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:19.265653 575 raft_consensus.cc:3037] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:19.266412 575 raft_consensus.cc:504] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.266603 575 leader_election.cc:288] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 1 election: Requested vote from peers 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115), 7997523c12664e30a31e70902792f240 (127.18.96.131:46743)
I20260928 14:50:19.266811 366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7796068140f64e05a9b6630b26afa131" candidate_uuid: "d3400aa1331c42da855345c407bf5eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa991f12e5943eeb335769d1e36c2db"
I20260928 14:50:19.266870 503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7796068140f64e05a9b6630b26afa131" candidate_uuid: "d3400aa1331c42da855345c407bf5eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7997523c12664e30a31e70902792f240"
I20260928 14:50:19.266896 366 raft_consensus.cc:3037] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:19.266964 503 raft_consensus.cc:3037] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:19.267635 366 raft_consensus.cc:2454] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3400aa1331c42da855345c407bf5eba in term 1.
I20260928 14:50:19.267748 503 raft_consensus.cc:2454] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3400aa1331c42da855345c407bf5eba in term 1.
I20260928 14:50:19.267834 32635 leader_election.cc:302] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa991f12e5943eeb335769d1e36c2db, d3400aa1331c42da855345c407bf5eba; no voters:
I20260928 14:50:19.267947 575 raft_consensus.cc:2781] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:19.268105 575 raft_consensus.cc:686] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 1 LEADER]: Becoming Leader. State: Replica: d3400aa1331c42da855345c407bf5eba, State: Running, Role: LEADER
I20260928 14:50:19.268216 575 consensus_queue.cc:237] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.269098 32553 catalog_manager.cc:5495] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba reported cstate change: term changed from 0 to 1, leader changed from <none> to d3400aa1331c42da855345c407bf5eba (127.18.96.129). New cstate: current_term: 1 leader_uuid: "d3400aa1331c42da855345c407bf5eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } health_report { overall_health: HEALTHY } } }
W20260928 14:50:19.275625 550 tablet.cc:2369] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:19.294035 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.291628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.18.96.129:36059)
0928 14:50:19.291689 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.291691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.291699 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:50:19.291785 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.291786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.291805 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.291901 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:19.292055 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.292568 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.293293 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.293480 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.293538 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.293667 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:19.293775 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:19.293836 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.293867 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.293886 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:50:19.296038 503 raft_consensus.cc:1261] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [term 1 FOLLOWER]: Refusing update from remote peer d3400aa1331c42da855345c407bf5eba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:19.296074 366 raft_consensus.cc:1261] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [term 1 FOLLOWER]: Refusing update from remote peer d3400aa1331c42da855345c407bf5eba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:19.296370 575 consensus_queue.cc:1035] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:19.296574 578 consensus_queue.cc:1035] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:19.299064 582 mvcc.cc:205] Tried to move back new op lower bound from 7334326351033393152 to 7334326350923300864. Current Snapshot: MvccSnapshot[applied={T|T < 7334326351033393152}]
I20260928 14:50:19.301440 32553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:19.301538 32553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:19.302371 32553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Sending DeleteTablet for 3 replicas of tablet 7796068140f64e05a9b6630b26afa131
I20260928 14:50:19.302707 346 tablet_service.cc:1514] Processing DeleteTablet for tablet 7796068140f64e05a9b6630b26afa131 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:19 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:36462
I20260928 14:50:19.302735 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 32616
I20260928 14:50:19.303464 483 tablet_service.cc:1514] Processing DeleteTablet for tablet 7796068140f64e05a9b6630b26afa131 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:19 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:57060
I20260928 14:50:19.303762 590 tablet_replica.cc:330] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: stopping tablet replica
I20260928 14:50:19.303907 590 raft_consensus.cc:2229] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:19.304021 590 raft_consensus.cc:2258] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:19.304174 591 tablet_replica.cc:330] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: stopping tablet replica
I20260928 14:50:19.304262 591 raft_consensus.cc:2229] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:19.304351 591 raft_consensus.cc:2258] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:19.304525 590 ts_tablet_manager.cc:1905] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:19.306156 590 ts_tablet_manager.cc:1918] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:19.306239 590 log.cc:1196] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/7796068140f64e05a9b6630b26afa131
I20260928 14:50:19.306444 590 ts_tablet_manager.cc:1939] T 7796068140f64e05a9b6630b26afa131 P 5fa991f12e5943eeb335769d1e36c2db: Deleting consensus metadata
I20260928 14:50:19.306840 32540 catalog_manager.cc:4841] TS 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115): tablet 7796068140f64e05a9b6630b26afa131 (table test-table [id=7537b1e7a6394af4a6642fae25a2a936]) successfully deleted
I20260928 14:50:19.308084 591 ts_tablet_manager.cc:1905] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:50:19.308148 32540 connection.cc:535] client connection to 127.18.96.129:36059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 14:50:19.308244 32540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:50:19.308431 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 32751
I20260928 14:50:19.309693 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.309568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36059 (local address 127.0.0.1:39970)
0928 14:50:19.309617 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.309650 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 14:50:19.309849 32540 catalog_manager.cc:4584] TS d3400aa1331c42da855345c407bf5eba (127.18.96.129:36059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7796068140f64e05a9b6630b26afa131: Network error: Client connection negotiation failed: client connection to 127.18.96.129:36059: connect: Connection refused (error 111)
I20260928 14:50:19.310014 591 ts_tablet_manager.cc:1918] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:19.310088 591 log.cc:1196] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/7796068140f64e05a9b6630b26afa131
I20260928 14:50:19.310369 591 ts_tablet_manager.cc:1939] T 7796068140f64e05a9b6630b26afa131 P 7997523c12664e30a31e70902792f240: Deleting consensus metadata
I20260928 14:50:19.310765 32540 catalog_manager.cc:4841] TS 7997523c12664e30a31e70902792f240 (127.18.96.131:46743): tablet 7796068140f64e05a9b6630b26afa131 (table test-table [id=7537b1e7a6394af4a6642fae25a2a936]) successfully deleted
I20260928 14:50:19.313958 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 419
I20260928 14:50:19.319670 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 32521
I20260928 14:50:19.325023 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43163
--webserver_interface=127.18.96.190
--webserver_port=33365
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:43163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:19.407393 593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:19.407655 593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:19.407691 593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:19.408898 593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:19.408977 593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:19.408999 593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:19.409019 593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:19.409050 593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:19.410530 593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:43163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:19.410806 593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:19.411039 593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:19.420771 600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:19.420981 593 server_base.cc:1034] running on GCE node
W20260928 14:50:19.420773 599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:19.420771 602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:19.421448 593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:19.421746 593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:19.422890 593 hybrid_clock.cc:648] HybridClock initialized: now 1790607019422880 us; error 33 us; skew 500 ppm
Sep 28 14:50:19 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:19.429448 593 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:50:19.429948 593 webserver.cc:460] Webserver started at http://127.18.96.190:33365/ using document root <none> and password file <none>
I20260928 14:50:19.430218 593 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:19.430279 593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:19.431820 593 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:50:19.432615 609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.432821 593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:19.432926 593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "75074b08af9943de801717c7f5c975e4"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "b829726da2a5547fe13a67606c27bbb4"
server_key_iv: "a1789a6e9b9a443a9882f25861d037e1"
server_key_version: "encryptionkey@0"
I20260928 14:50:19.433357 593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:19.449507 593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:19.452046 593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:19.452207 593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:19.456487 593 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:43163
I20260928 14:50:19.456524 661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:43163 every 8 connection(s)
I20260928 14:50:19.456916 593 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:19.458532 662 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:50:19.459143 662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Bootstrap starting.
I20260928 14:50:19.460180 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 593
I20260928 14:50:19.460457 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:36059
--local_ip_for_outbound_sockets=127.18.96.129
--tserver_master_addrs=127.18.96.190:43163
--webserver_port=35751
--webserver_interface=127.18.96.129
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:19.468348 662 log.cc:824] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:19.470443 662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:50:19.470769 662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Bootstrap complete.
I20260928 14:50:19.473353 662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:19.473603 662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75074b08af9943de801717c7f5c975e4, State: Initialized, Role: FOLLOWER
I20260928 14:50:19.473961 662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:19.474100 662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:19.474203 662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:19.474295 662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:50:19.475418 662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:19.475513 662 leader_election.cc:302] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75074b08af9943de801717c7f5c975e4; no voters:
I20260928 14:50:19.475747 662 leader_election.cc:288] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:50:19.475790 666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:50:19.475951 666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 2 LEADER]: Becoming Leader. State: Replica: 75074b08af9943de801717c7f5c975e4, State: Running, Role: LEADER
I20260928 14:50:19.476070 662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:19.476044 666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:19.476457 667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "75074b08af9943de801717c7f5c975e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } } }
I20260928 14:50:19.476470 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75074b08af9943de801717c7f5c975e4. Latest consensus state: current_term: 2 leader_uuid: "75074b08af9943de801717c7f5c975e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } } }
I20260928 14:50:19.476635 667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:19.476655 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:19.477058 675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:19.477933 675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:19.478042 675 catalog_manager.cc:1259] Loaded cluster ID: 8cb3e024d27b4fdd8497975ae47ec906
I20260928 14:50:19.478081 675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:19.478509 675 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 14:50:19.481348 675 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 14:50:19.483246 675 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:19.483466 675 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Loaded TSK: 0
I20260928 14:50:19.483749 675 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:50:19.558082 664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:19.558333 664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:19.558377 664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:19.559588 664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:19.559662 664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:19.559707 664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:50:19.561257 664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:36059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=35751
--tserver_master_addrs=127.18.96.190:43163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:19.561524 664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:19.561770 664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:19.567999 692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:19.568087 689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:19.567965 690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:19.569058 664 server_base.cc:1034] running on GCE node
I20260928 14:50:19.569272 664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:19.569550 664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:19.570719 664 hybrid_clock.cc:648] HybridClock initialized: now 1790607019570686 us; error 48 us; skew 500 ppm
Sep 28 14:50:19 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:19.573523 664 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:50:19.574062 664 webserver.cc:460] Webserver started at http://127.18.96.129:35751/ using document root <none> and password file <none>
I20260928 14:50:19.574326 664 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:19.574388 664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:19.575975 664 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:50:19.576845 700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.577039 664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:19.577147 664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "d3400aa1331c42da855345c407bf5eba"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "02e8ae15200cf517ef6fbad5e02036dd"
server_key_iv: "fcaa3f47b673c1eefdb63759a45f9dcb"
server_key_version: "encryptionkey@0"
I20260928 14:50:19.577684 664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:19.601578 664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:19.604249 664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:19.604406 664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:19.604672 664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:19.605207 707 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:50:19.606752 664 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:50:19.606817 664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 14:50:19.606853 664 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:50:19.607748 664 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:50:19.607795 664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 14:50:19.607869 707 tablet_bootstrap.cc:492] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Bootstrap starting.
I20260928 14:50:19.615224 664 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:36059
I20260928 14:50:19.615690 664 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:19.616080 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 664
I20260928 14:50:19.616417 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:33115
--local_ip_for_outbound_sockets=127.18.96.130
--tserver_master_addrs=127.18.96.190:43163
--webserver_port=46377
--webserver_interface=127.18.96.130
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:19.618918 707 log.cc:824] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:19.619354 814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:36059 every 8 connection(s)
I20260928 14:50:19.622004 707 tablet_bootstrap.cc:492] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:50:19.622303 707 tablet_bootstrap.cc:492] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Bootstrap complete.
I20260928 14:50:19.623596 707 ts_tablet_manager.cc:1397] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260928 14:50:19.625423 707 raft_consensus.cc:348] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.625756 707 raft_consensus.cc:729] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3400aa1331c42da855345c407bf5eba, State: Initialized, Role: FOLLOWER
I20260928 14:50:19.625890 707 consensus_queue.cc:260] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.626179 707 ts_tablet_manager.cc:1428] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Time spent starting tablet: real 0.003s user 0.003s sys 0.003s
Sep 28 14:50:19 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:19.638118 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.620015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:54875)
0928 14:50:19.623998 (+ 3983us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.624055 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.625429 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.626451 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.626456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.626859 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:19.627160 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.627167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.628999 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.629001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.630391 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.630395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.631199 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.631201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.631361 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.631982 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:19.631992 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:19.633839 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:19.636204 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.636211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.636213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.636526 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.636810 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.636811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.636811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.636952 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.637463 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:19.637470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:19.637626 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.637737 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.637907 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3931,"thread_start_us":92,"threads_started":1}
I20260928 14:50:19.638139 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.620061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:54875 (local address 127.18.96.190:43163)
0928 14:50:19.623850 (+ 3789us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.623863 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.624209 (+ 346us) server_negotiation.cc:407] Connection header received
0928 14:50:19.625711 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.625720 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.626060 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.626207 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:19.627303 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.628876 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.630497 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.631059 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.631192 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.633953 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:19.633958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:19.633958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:19.633973 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:19.636074 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.636614 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.636617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.636617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.636737 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.637027 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.637028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.637029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.637387 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:19.637470 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.637806 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.637948 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3718,"thread_start_us":82,"threads_started":1}
I20260928 14:50:19.639256 815 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43163
I20260928 14:50:19.639367 815 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:19.639562 815 heartbeater.cc:511] Master 127.18.96.190:43163 requested a full tablet report, sending...
I20260928 14:50:19.640391 626 ts_manager.cc:194] Registered new tserver with Master: d3400aa1331c42da855345c407bf5eba (127.18.96.129:36059)
I20260928 14:50:19.641630 626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:54875
I20260928 14:50:19.642741 815 heartbeater.cc:503] Master 127.18.96.190:43163 was elected leader, sending a full tablet report...
Sep 28 14:50:19 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:50:19.650838 822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.641159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36059 (local address 127.0.0.1:39974)
0928 14:50:19.642050 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.642090 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.642197 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.642396 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.642401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.642770 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:19.642852 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.642855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.644306 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.644307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.644946 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.644949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.645605 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.645607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.645685 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.646261 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:19.646265 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:19.647695 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:19.649622 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.649629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.649630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.649828 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.650091 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.650092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.650093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.650165 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.650495 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:19.650503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:19.650557 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.650639 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.650757 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":117,"threads_started":1}
I20260928 14:50:19.650873 823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.641219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.18.96.129:36059)
0928 14:50:19.641965 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.641970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.642174 (+ 204us) server_negotiation.cc:407] Connection header received
0928 14:50:19.642289 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.642291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.642312 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.642391 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:19.643137 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.644201 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.645027 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.645492 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.645564 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.647790 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:19.647800 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:19.647800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:19.647809 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:19.649533 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.649911 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.649913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.649914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.649981 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.650239 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.650240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.650240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.650401 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:19.650481 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.650689 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.650777 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":88,"threads_started":1}
I20260928 14:50:19.651301 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 7796068140f64e05a9b6630b26afa131 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:19 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:39974
I20260928 14:50:19.651940 824 tablet_replica.cc:330] stopping tablet replica
I20260928 14:50:19.652046 824 raft_consensus.cc:2229] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:19.652117 824 raft_consensus.cc:2258] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:19.652453 824 ts_tablet_manager.cc:1905] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:19.654773 824 ts_tablet_manager.cc:1918] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:19.654853 824 log.cc:1196] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/7796068140f64e05a9b6630b26afa131
I20260928 14:50:19.655083 824 ts_tablet_manager.cc:1939] T 7796068140f64e05a9b6630b26afa131 P d3400aa1331c42da855345c407bf5eba: Deleting consensus metadata
I20260928 14:50:19.655668 613 catalog_manager.cc:4841] TS d3400aa1331c42da855345c407bf5eba (127.18.96.129:36059): tablet 7796068140f64e05a9b6630b26afa131 (table test-table [id=7537b1e7a6394af4a6642fae25a2a936]) successfully deleted
W20260928 14:50:19.710294 817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:19.710503 817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:19.710527 817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:19.711735 817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:19.711777 817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:19.711804 817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:50:19.713339 817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:33115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=46377
--tserver_master_addrs=127.18.96.190:43163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:19.713598 817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:19.713852 817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:19.719352 832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:19.719359 829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:19.719632 817 server_base.cc:1034] running on GCE node
W20260928 14:50:19.719367 830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:19.719933 817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:19.720201 817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:19.721361 817 hybrid_clock.cc:648] HybridClock initialized: now 1790607019721349 us; error 46 us; skew 500 ppm
Sep 28 14:50:19 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:19.723838 817 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:50:19.724373 817 webserver.cc:460] Webserver started at http://127.18.96.130:46377/ using document root <none> and password file <none>
I20260928 14:50:19.724622 817 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:19.724717 817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:19.726079 817 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:19.726735 839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.726935 817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:19.727037 817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "5fa991f12e5943eeb335769d1e36c2db"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "17c3c08b22f950f0a0a19240fd1c7cdc"
server_key_iv: "a0dfce3c26b0dc75d42b9eee14c019cb"
server_key_version: "encryptionkey@0"
I20260928 14:50:19.727401 817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:19.748824 817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:19.751223 817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:19.751417 817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:19.751680 817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:19.752053 817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:19.752110 817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.752147 817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:19.752197 817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.758665 817 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:33115
I20260928 14:50:19.758718 953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:33115 every 8 connection(s)
I20260928 14:50:19.759069 817 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:19.763185 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 817
I20260928 14:50:19.763520 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:46743
--local_ip_for_outbound_sockets=127.18.96.131
--tserver_master_addrs=127.18.96.190:43163
--webserver_port=36341
--webserver_interface=127.18.96.131
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:50:19 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:19.769905 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.759853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:39029 (local address 127.18.96.190:43163)
0928 14:50:19.759930 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.759933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.760187 (+ 254us) server_negotiation.cc:407] Connection header received
0928 14:50:19.760740 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.760741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.760758 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.760810 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:19.761809 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.762340 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.763511 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.763847 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.763975 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.766731 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:19.766732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:19.766732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:19.766741 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:19.768259 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.768835 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.768836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.768837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.768896 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.769213 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.769214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.769214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.769325 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:19.769378 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.769755 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.769836 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:50:19.770244 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.759858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:39029)
0928 14:50:19.760042 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.760081 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.760623 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.760891 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.760895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.761491 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:19.761701 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.761705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.762466 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.762467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.763417 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.763420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.763989 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.763990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.764121 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.764709 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:19.764718 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:19.766609 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:19.768346 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.768356 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.768359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.768726 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.768974 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.768975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.768976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.769137 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.769395 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:19.769402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:19.769604 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.769739 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.769942 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260928 14:50:19.770852 954 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43163
I20260928 14:50:19.770951 954 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:19.771162 954 heartbeater.cc:511] Master 127.18.96.190:43163 requested a full tablet report, sending...
I20260928 14:50:19.771616 626 ts_manager.cc:194] Registered new tserver with Master: 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115)
I20260928 14:50:19.772069 626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:39029
W20260928 14:50:19.849874 957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:19.850137 957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:19.850180 957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:19.851420 957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:19.851493 957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:19.851541 957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:50:19.853163 957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:46743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=36341
--tserver_master_addrs=127.18.96.190:43163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:19.853436 957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:19.853669 957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:19.858873 965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:19.858879 962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:19.858886 963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:19.859119 957 server_base.cc:1034] running on GCE node
I20260928 14:50:19.859364 957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:19.859572 957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:19.860740 957 hybrid_clock.cc:648] HybridClock initialized: now 1790607019860718 us; error 41 us; skew 500 ppm
Sep 28 14:50:19 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:19.863341 957 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:50:19.863785 957 webserver.cc:460] Webserver started at http://127.18.96.131:36341/ using document root <none> and password file <none>
I20260928 14:50:19.864053 957 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:19.864137 957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:19.865504 957 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:19.866137 972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.866326 957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:19.866429 957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "7997523c12664e30a31e70902792f240"
format_stamp: "Formatted at 2026-09-28 14:50:19 on dist-test-slave-j968"
server_key: "cfe741fcb5059e4be27d6732558227c9"
server_key_iv: "0fc88a765341e6f7f5642c4f73ac95a7"
server_key_version: "encryptionkey@0"
I20260928 14:50:19.866770 957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:19.895174 957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:19.897389 957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:19.897555 957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:19.897799 957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:19.898119 957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:19.898185 957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.898231 957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:19.898252 957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:19.904073 957 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:46743
I20260928 14:50:19.904140 1086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:46743 every 8 connection(s)
I20260928 14:50:19.904512 957 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:19.909977 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 957
Sep 28 14:50:19 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607019, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:19.915695 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.905713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:46915 (local address 127.18.96.190:43163)
0928 14:50:19.905808 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.905810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.906070 (+ 260us) server_negotiation.cc:407] Connection header received
0928 14:50:19.906605 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.906606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.906618 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.906689 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:19.907778 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.908412 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.909572 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.909777 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.909837 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.912584 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:19.912586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:19.912587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:19.912594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:19.914173 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.914620 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.914621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.914622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.914683 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.914943 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.914944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.914945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.915072 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:19.915163 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.915519 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.915627 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:50:19.915946 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.905717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:46915)
0928 14:50:19.905894 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.905942 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.906471 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.906748 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.906751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.907448 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:19.907665 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.907670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.908509 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.908510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.909446 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.909449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.909882 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.909883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.910179 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.910938 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:19.910948 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:19.912484 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:19.914286 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.914294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.914296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.914528 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.914778 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:19.914779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:19.914780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:19.914891 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:19.915152 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:19.915160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:19.915326 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.915440 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.915630 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260928 14:50:19.916539 1087 heartbeater.cc:347] Connected to a master server at 127.18.96.190:43163
I20260928 14:50:19.916625 1087 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:19.916844 1087 heartbeater.cc:511] Master 127.18.96.190:43163 requested a full tablet report, sending...
I20260928 14:50:19.917335 626 ts_manager.cc:194] Registered new tserver with Master: 7997523c12664e30a31e70902792f240 (127.18.96.131:46743)
I20260928 14:50:19.917603 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.910532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.18.96.190:43163)
0928 14:50:19.910688 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.910690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.910746 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:50:19.910828 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.910828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.910840 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.910924 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:19.911563 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.912107 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.912841 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.913035 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.913116 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.914449 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:19.914450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:19.914451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:19.914458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:19.916356 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.916771 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.916772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.916773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.916832 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:19.917037 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:19.917038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:19.917038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:19.917152 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:19.917344 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.917446 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.917533 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260928 14:50:19.917896 626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:46915
I20260928 14:50:19.918347 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:19.921272 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.918709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.18.96.190:43163)
0928 14:50:19.918742 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.918745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.918860 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:50:19.918952 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.918953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.918989 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.919065 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:19.919165 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.919642 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.920468 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.920691 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.920767 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.920935 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:19.921044 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:19.921130 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.921179 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.921201 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 14:50:19.921525 626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:19.923036 626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:19.925702 749 tablet_service.cc:1467] Processing CreateTablet for tablet ba85c8569f3a44f2aa0cc93a34cd5abe (DEFAULT_TABLE table=test-table [id=deebc7389c53448fa6818156874c2a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:19.925909 749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba85c8569f3a44f2aa0cc93a34cd5abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:19.927250 707 tablet_bootstrap.cc:492] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Bootstrap starting.
I20260928 14:50:19.927582 707 tablet_bootstrap.cc:654] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:19.928299 707 tablet_bootstrap.cc:492] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: No bootstrap required, opened a new log
I20260928 14:50:19.928347 707 ts_tablet_manager.cc:1397] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:50:19.928643 707 raft_consensus.cc:348] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.928700 707 raft_consensus.cc:374] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:19.928728 707 raft_consensus.cc:729] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3400aa1331c42da855345c407bf5eba, State: Initialized, Role: FOLLOWER
I20260928 14:50:19.928791 707 consensus_queue.cc:260] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.928884 707 ts_tablet_manager.cc:1428] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:50:19.929345 822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.925065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:33115 (local address 127.0.0.1:36476)
0928 14:50:19.925117 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.925128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.925166 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.925635 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.925636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.925637 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:19.925698 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.925699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.927029 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.927030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.928247 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.928251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.929081 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.929082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.929153 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.929173 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.929206 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.929260 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":54}
I20260928 14:50:19.929443 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.925458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.18.96.131:46743)
0928 14:50:19.925624 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.925628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.925637 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:50:19.925671 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.925673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.925687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.925758 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:19.925966 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.927008 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.928260 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.928977 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.929053 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.929115 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.929292 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.929345 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260928 14:50:19.929347 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.925174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:46743 (local address 127.0.0.1:57068)
0928 14:50:19.925332 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:19.925343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:19.925397 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:19.925836 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:19.925837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:19.925839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:19.925877 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.925878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.927089 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.927090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.928164 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:19.928167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.929145 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.929145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:19.929194 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.929198 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:50:19.929253 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:50:19.929282 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260928 14:50:19.929780 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:19.925198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.18.96.130:33115)
0928 14:50:19.925422 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:19.925428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:19.925447 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:50:19.925490 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:19.925492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:19.925508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:19.925623 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:19.925763 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.926911 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.928322 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:19.928991 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:19.929533 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:19.929608 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:19.929635 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 14:50:19.929679 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":116,"threads_started":1}
I20260928 14:50:19.930517 1021 tablet_service.cc:1467] Processing CreateTablet for tablet ba85c8569f3a44f2aa0cc93a34cd5abe (DEFAULT_TABLE table=test-table [id=deebc7389c53448fa6818156874c2a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:19.930580 888 tablet_service.cc:1467] Processing CreateTablet for tablet ba85c8569f3a44f2aa0cc93a34cd5abe (DEFAULT_TABLE table=test-table [id=deebc7389c53448fa6818156874c2a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:19.930877 888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba85c8569f3a44f2aa0cc93a34cd5abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:19.930878 1021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba85c8569f3a44f2aa0cc93a34cd5abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:19.934541 1097 tablet_bootstrap.cc:492] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: Bootstrap starting.
I20260928 14:50:19.934762 1098 tablet_bootstrap.cc:492] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: Bootstrap starting.
I20260928 14:50:19.935360 1097 tablet_bootstrap.cc:654] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:19.935360 1098 tablet_bootstrap.cc:654] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:19.936056 1098 log.cc:824] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:19.936126 1097 log.cc:824] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:19.936817 1097 tablet_bootstrap.cc:492] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: No bootstrap required, opened a new log
I20260928 14:50:19.936903 1097 ts_tablet_manager.cc:1397] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:50:19.937232 1098 tablet_bootstrap.cc:492] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: No bootstrap required, opened a new log
I20260928 14:50:19.937310 1098 ts_tablet_manager.cc:1397] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:50:19.938901 1097 raft_consensus.cc:348] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.939078 1097 raft_consensus.cc:374] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:19.939121 1097 raft_consensus.cc:729] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7997523c12664e30a31e70902792f240, State: Initialized, Role: FOLLOWER
I20260928 14:50:19.939261 1097 consensus_queue.cc:260] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.939260 1098 raft_consensus.cc:348] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.939424 1098 raft_consensus.cc:374] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:19.939471 1098 raft_consensus.cc:729] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa991f12e5943eeb335769d1e36c2db, State: Initialized, Role: FOLLOWER
I20260928 14:50:19.939560 1097 ts_tablet_manager.cc:1428] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:50:19.939586 1098 consensus_queue.cc:260] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:19.939620 1087 heartbeater.cc:503] Master 127.18.96.190:43163 was elected leader, sending a full tablet report...
I20260928 14:50:19.939924 1098 ts_tablet_manager.cc:1428] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:19.940013 954 heartbeater.cc:503] Master 127.18.96.190:43163 was elected leader, sending a full tablet report...
W20260928 14:50:20.009773 955 tablet.cc:2369] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:50:20.120189 816 tablet.cc:2369] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:50:20.155411 1088 tablet.cc:2369] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:20.182163 1101 raft_consensus.cc:482] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:20.182344 1101 raft_consensus.cc:504] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:20.182785 1101 leader_election.cc:288] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115), d3400aa1331c42da855345c407bf5eba (127.18.96.129:36059)
I20260928 14:50:20.186658 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.182988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:33115 (local address 127.18.96.131:44883)
0928 14:50:20.183063 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.183079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:20.183153 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:20.183350 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:20.183352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:20.183354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:20.183430 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:20.183433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.184435 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.184436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:20.185593 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:20.185595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.186430 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.186430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:20.186490 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:20.186493 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:50:20.186535 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 14:50:20.186570 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 14:50:20.186688 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.183070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:44883 (local address 127.18.96.130:33115)
0928 14:50:20.183121 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:20.183123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:20.183136 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:50:20.183240 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:20.183241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:20.183254 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:20.183307 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:20.183515 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.184356 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.185699 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.186345 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.186431 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:20.186440 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:20.186568 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 14:50:20.186618 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 14:50:20.186940 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.183107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36059 (local address 127.18.96.131:36621)
0928 14:50:20.183306 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.183316 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:20.183373 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:20.183604 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:20.183605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:20.183607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:20.183692 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:20.183694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.184656 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.184656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:20.185753 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:20.185754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.186659 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.186660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:20.186753 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:20.186758 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:50:20.186809 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:50:20.186854 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
I20260928 14:50:20.187037 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba85c8569f3a44f2aa0cc93a34cd5abe" candidate_uuid: "7997523c12664e30a31e70902792f240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa991f12e5943eeb335769d1e36c2db" is_pre_election: true
I20260928 14:50:20.187134 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.183193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:36621 (local address 127.18.96.129:36059)
0928 14:50:20.183310 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:20.183313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:20.183369 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:50:20.183428 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:20.183429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:20.183449 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:20.183557 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:20.183785 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.184583 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.185842 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.186573 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.186648 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:20.186907 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:20.186928 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 14:50:20.186978 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
I20260928 14:50:20.187229 908 raft_consensus.cc:2454] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7997523c12664e30a31e70902792f240 in term 0.
I20260928 14:50:20.187501 976 leader_election.cc:302] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa991f12e5943eeb335769d1e36c2db, 7997523c12664e30a31e70902792f240; no voters:
I20260928 14:50:20.187564 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba85c8569f3a44f2aa0cc93a34cd5abe" candidate_uuid: "7997523c12664e30a31e70902792f240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3400aa1331c42da855345c407bf5eba" is_pre_election: true
I20260928 14:50:20.187655 1101 raft_consensus.cc:2781] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:20.187713 1101 raft_consensus.cc:482] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:20.187736 769 raft_consensus.cc:2454] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7997523c12664e30a31e70902792f240 in term 0.
I20260928 14:50:20.187750 1101 raft_consensus.cc:3037] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:20.188583 1101 raft_consensus.cc:504] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:20.188735 1101 leader_election.cc:288] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [CANDIDATE]: Term 1 election: Requested vote from peers 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115), d3400aa1331c42da855345c407bf5eba (127.18.96.129:36059)
I20260928 14:50:20.188915 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba85c8569f3a44f2aa0cc93a34cd5abe" candidate_uuid: "7997523c12664e30a31e70902792f240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3400aa1331c42da855345c407bf5eba"
I20260928 14:50:20.188928 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba85c8569f3a44f2aa0cc93a34cd5abe" candidate_uuid: "7997523c12664e30a31e70902792f240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa991f12e5943eeb335769d1e36c2db"
I20260928 14:50:20.189008 769 raft_consensus.cc:3037] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:20.189011 908 raft_consensus.cc:3037] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:20.189695 769 raft_consensus.cc:2454] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7997523c12664e30a31e70902792f240 in term 1.
I20260928 14:50:20.189826 908 raft_consensus.cc:2454] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7997523c12664e30a31e70902792f240 in term 1.
I20260928 14:50:20.189944 976 leader_election.cc:302] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7997523c12664e30a31e70902792f240, d3400aa1331c42da855345c407bf5eba; no voters:
I20260928 14:50:20.190093 1101 raft_consensus.cc:2781] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:20.190234 1101 raft_consensus.cc:686] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 1 LEADER]: Becoming Leader. State: Replica: 7997523c12664e30a31e70902792f240, State: Running, Role: LEADER
I20260928 14:50:20.190340 1101 consensus_queue.cc:237] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:20.191212 625 catalog_manager.cc:5495] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7997523c12664e30a31e70902792f240 (127.18.96.131). New cstate: current_term: 1 leader_uuid: "7997523c12664e30a31e70902792f240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } health_report { overall_health: UNKNOWN } } }
I20260928 14:50:20.214226 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.211737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.18.96.131:46743)
0928 14:50:20.211799 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:20.211801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:20.211817 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:50:20.211854 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:20.211854 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:20.211927 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:20.211997 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:20.212144 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.212651 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.213468 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.213647 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.213699 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:20.213867 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:20.213970 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:20.214032 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:20.214078 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 14:50:20.214095 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:50:20.216318 769 raft_consensus.cc:1261] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Refusing update from remote peer 7997523c12664e30a31e70902792f240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:20.216707 1105 consensus_queue.cc:1035] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:20.216786 908 raft_consensus.cc:1261] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [term 1 FOLLOWER]: Refusing update from remote peer 7997523c12664e30a31e70902792f240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:20.217088 1105 consensus_queue.cc:1035] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:20.219765 1109 mvcc.cc:205] Tried to move back new op lower bound from 7334326354802139136 to 7334326354700275712. Current Snapshot: MvccSnapshot[applied={T|T < 7334326354802139136}]
I20260928 14:50:20.220294 1111 mvcc.cc:205] Tried to move back new op lower bound from 7334326354802139136 to 7334326354700275712. Current Snapshot: MvccSnapshot[applied={T|T < 7334326354802139136}]
I20260928 14:50:20.221418 625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:20.221514 625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:20.222318 625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Sending DeleteTablet for 3 replicas of tablet ba85c8569f3a44f2aa0cc93a34cd5abe
I20260928 14:50:20.222664 749 tablet_service.cc:1514] Processing DeleteTablet for tablet ba85c8569f3a44f2aa0cc93a34cd5abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:20 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:39974
I20260928 14:50:20.222664 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet ba85c8569f3a44f2aa0cc93a34cd5abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:20 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:57068
I20260928 14:50:20.222711 888 tablet_service.cc:1514] Processing DeleteTablet for tablet ba85c8569f3a44f2aa0cc93a34cd5abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:20 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:36476
I20260928 14:50:20.222890 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 664
I20260928 14:50:20.222929 1120 tablet_replica.cc:330] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: stopping tablet replica
I20260928 14:50:20.224027 1118 tablet_replica.cc:330] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: stopping tablet replica
I20260928 14:50:20.224131 1118 raft_consensus.cc:2229] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:50:20.224238 1118 raft_consensus.cc:2258] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:20.224694 1118 ts_tablet_manager.cc:1905] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:20.224987 1120 raft_consensus.cc:2229] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:20.225140 1120 raft_consensus.cc:2258] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:20.225589 1120 ts_tablet_manager.cc:1905] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:20.226425 1118 ts_tablet_manager.cc:1918] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:20.226549 1118 log.cc:1196] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/ba85c8569f3a44f2aa0cc93a34cd5abe
I20260928 14:50:20.226902 1118 ts_tablet_manager.cc:1939] T ba85c8569f3a44f2aa0cc93a34cd5abe P 7997523c12664e30a31e70902792f240: Deleting consensus metadata
I20260928 14:50:20.227280 1120 ts_tablet_manager.cc:1918] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:20.227350 1120 log.cc:1196] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/ba85c8569f3a44f2aa0cc93a34cd5abe
I20260928 14:50:20.227386 613 catalog_manager.cc:4841] TS 7997523c12664e30a31e70902792f240 (127.18.96.131:46743): tablet ba85c8569f3a44f2aa0cc93a34cd5abe (table test-table [id=deebc7389c53448fa6818156874c2a29]) successfully deleted
I20260928 14:50:20.227595 1120 ts_tablet_manager.cc:1939] T ba85c8569f3a44f2aa0cc93a34cd5abe P 5fa991f12e5943eeb335769d1e36c2db: Deleting consensus metadata
I20260928 14:50:20.228004 613 catalog_manager.cc:4841] TS 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115): tablet ba85c8569f3a44f2aa0cc93a34cd5abe (table test-table [id=deebc7389c53448fa6818156874c2a29]) successfully deleted
W20260928 14:50:20.229981 613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:36059 (error 108)
I20260928 14:50:20.230254 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 817
I20260928 14:50:20.230901 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.230392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:36059 (local address 127.0.0.1:39986)
0928 14:50:20.230820 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.230859 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408}
W20260928 14:50:20.231387 613 catalog_manager.cc:4584] TS d3400aa1331c42da855345c407bf5eba (127.18.96.129:36059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba85c8569f3a44f2aa0cc93a34cd5abe: Network error: Client connection negotiation failed: client connection to 127.18.96.129:36059: connect: Connection refused (error 111)
I20260928 14:50:20.236197 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 957
I20260928 14:50:20.241276 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 593
I20260928 14:50:20.246569 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43163
--webserver_interface=127.18.96.190
--webserver_port=33365
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:43163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:20.327884 1122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:20.328089 1122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:20.328119 1122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:20.329234 1122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:20.329272 1122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:20.329288 1122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:20.329299 1122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:20.329311 1122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:20.330724 1122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:43163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:20.330917 1122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:20.331090 1122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:20.335986 1130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:20.335980 1128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:20.335980 1127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:20.336108 1122 server_base.cc:1034] running on GCE node
I20260928 14:50:20.336454 1122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:20.336758 1122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:20.337924 1122 hybrid_clock.cc:648] HybridClock initialized: now 1790607020337915 us; error 48 us; skew 500 ppm
Sep 28 14:50:20 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607020, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:20.340417 1122 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:50:20.340869 1122 webserver.cc:460] Webserver started at http://127.18.96.190:33365/ using document root <none> and password file <none>
I20260928 14:50:20.341070 1122 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:20.341117 1122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:20.342429 1122 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:20.343044 1137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:20.343248 1122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:20.343354 1122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "75074b08af9943de801717c7f5c975e4"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "b829726da2a5547fe13a67606c27bbb4"
server_key_iv: "a1789a6e9b9a443a9882f25861d037e1"
server_key_version: "encryptionkey@0"
I20260928 14:50:20.343700 1122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:20.360711 1122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:20.362617 1122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:20.362779 1122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:20.366878 1122 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:43163
I20260928 14:50:20.366911 1191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:43163 every 8 connection(s)
I20260928 14:50:20.367338 1122 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:20.368920 1192 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:50:20.369585 1192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Bootstrap starting.
I20260928 14:50:20.371322 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 1122
I20260928 14:50:20.371640 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:36059
--local_ip_for_outbound_sockets=127.18.96.129
--tserver_master_addrs=127.18.96.190:43163
--webserver_port=35751
--webserver_interface=127.18.96.129
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:20.378518 1192 log.cc:824] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:20.380836 1192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:50:20.381137 1192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4: Bootstrap complete.
I20260928 14:50:20.383417 1192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:20.383669 1192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75074b08af9943de801717c7f5c975e4, State: Initialized, Role: FOLLOWER
I20260928 14:50:20.383934 1192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:20.384037 1192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:20.384091 1192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:20.384140 1192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:50:20.385126 1192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:20.385253 1192 leader_election.cc:302] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75074b08af9943de801717c7f5c975e4; no voters:
I20260928 14:50:20.385435 1192 leader_election.cc:288] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:50:20.385506 1196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:50:20.385646 1196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [term 3 LEADER]: Becoming Leader. State: Replica: 75074b08af9943de801717c7f5c975e4, State: Running, Role: LEADER
I20260928 14:50:20.385718 1192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:20.385816 1196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } }
I20260928 14:50:20.386132 1197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "75074b08af9943de801717c7f5c975e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } } }
I20260928 14:50:20.386132 1196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75074b08af9943de801717c7f5c975e4. Latest consensus state: current_term: 3 leader_uuid: "75074b08af9943de801717c7f5c975e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75074b08af9943de801717c7f5c975e4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43163 } } }
I20260928 14:50:20.386294 1197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:20.386338 1196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75074b08af9943de801717c7f5c975e4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:20.387254 1203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:20.388073 1203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:20.388203 1203 catalog_manager.cc:1259] Loaded cluster ID: 8cb3e024d27b4fdd8497975ae47ec906
I20260928 14:50:20.388260 1203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 14:50:20.388505 1203 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4be40979ad google::LogMessage::Fail() at ??:0
@ 0x7f4be409a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4be40974de google::LogMessage::Flush() at ??:0
@ 0x7f4be4098a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4bebfb80f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f4bebfc18b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4be5f650d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4be5f5a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4be6dbe6db start_thread at ??:0
@ 0x7f4be2b9561f clone at ??:0
W20260928 14:50:20.472445 1194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:20.472646 1194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:20.472671 1194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:20.474602 1194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:20.474678 1194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:20.474725 1194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:50:20.476744 1194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:36059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=35751
--tserver_master_addrs=127.18.96.190:43163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:20.477020 1194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:20.477388 1194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:20.482653 1228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:20.482749 1194 server_base.cc:1034] running on GCE node
W20260928 14:50:20.482653 1229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:20.482813 1231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:20.483144 1194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:20.483409 1194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:20.484581 1194 hybrid_clock.cc:648] HybridClock initialized: now 1790607020484563 us; error 32 us; skew 500 ppm
Sep 28 14:50:20 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607020, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:20.487144 1194 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:50:20.487648 1194 webserver.cc:460] Webserver started at http://127.18.96.129:35751/ using document root <none> and password file <none>
I20260928 14:50:20.487984 1194 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:20.488035 1194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:20.489382 1194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:20.490557 1238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:20.490761 1194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:20.490861 1194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "d3400aa1331c42da855345c407bf5eba"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "02e8ae15200cf517ef6fbad5e02036dd"
server_key_iv: "fcaa3f47b673c1eefdb63759a45f9dcb"
server_key_version: "encryptionkey@0"
I20260928 14:50:20.491384 1194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:20.507907 1194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:20.510551 1194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:20.510722 1194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:20.510998 1194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:20.511556 1246 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:50:20.513252 1194 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:50:20.513317 1194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 14:50:20.513355 1194 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:50:20.514277 1194 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:50:20.514326 1194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 14:50:20.514420 1246 tablet_bootstrap.cc:492] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Bootstrap starting.
I20260928 14:50:20.522428 1194 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:36059
I20260928 14:50:20.522504 1354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:36059 every 8 connection(s)
I20260928 14:50:20.522943 1194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:20.525326 1246 log.cc:824] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:20.527033 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 1194
I20260928 14:50:20.527308 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:33115
--local_ip_for_outbound_sockets=127.18.96.130
--tserver_master_addrs=127.18.96.190:43163
--webserver_port=46377
--webserver_interface=127.18.96.130
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:20.527789 1246 tablet_bootstrap.cc:492] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:50:20.528129 1246 tablet_bootstrap.cc:492] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Bootstrap complete.
I20260928 14:50:20.529268 1246 ts_tablet_manager.cc:1397] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20260928 14:50:20.530334 1246 raft_consensus.cc:348] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:20.530620 1246 raft_consensus.cc:729] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3400aa1331c42da855345c407bf5eba, State: Initialized, Role: FOLLOWER
I20260928 14:50:20.530804 1246 consensus_queue.cc:260] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:20.531096 1246 ts_tablet_manager.cc:1428] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260928 14:50:20.583292 1357 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:20.523691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:34685)
0928 14:50:20.523969 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.524027 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:20.524710 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:20.583049 (+ 58339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":126,"threads_started":1}
W20260928 14:50:20.583612 1355 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:43163 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.190:43163: Client connection negotiation failed: client connection to 127.18.96.190:43163: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:50:20.583998 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.583855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:36413)
0928 14:50:20.583931 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.583972 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 14:50:20.584372 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.584283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:36643)
0928 14:50:20.584325 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.584350 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 14:50:20.584530 1355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:50:20.615947 1359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:20.616171 1359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:20.616210 1359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:20.617391 1359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:20.617456 1359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:20.617501 1359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:50:20.618940 1359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:33115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=46377
--tserver_master_addrs=127.18.96.190:43163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:20.619297 1359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:20.619537 1359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:20.624536 1365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:20.624539 1366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:20.624595 1368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:20.624842 1359 server_base.cc:1034] running on GCE node
I20260928 14:50:20.625034 1359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:20.625267 1359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:20.626410 1359 hybrid_clock.cc:648] HybridClock initialized: now 1790607020626396 us; error 28 us; skew 500 ppm
Sep 28 14:50:20 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607020, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:20.628880 1359 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:50:20.629294 1359 webserver.cc:460] Webserver started at http://127.18.96.130:46377/ using document root <none> and password file <none>
I20260928 14:50:20.629526 1359 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:20.629572 1359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:20.630879 1359 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:20.631492 1375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:20.631696 1359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:20.631785 1359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "5fa991f12e5943eeb335769d1e36c2db"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-j968"
server_key: "17c3c08b22f950f0a0a19240fd1c7cdc"
server_key_iv: "a0dfce3c26b0dc75d42b9eee14c019cb"
server_key_version: "encryptionkey@0"
I20260928 14:50:20.632164 1359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:20.642244 1359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:20.644209 1359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:20.644352 1359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:20.644578 1359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:20.644915 1359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:20.644968 1359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:20.645008 1359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:20.645033 1359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:20.650907 1359 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:33115
I20260928 14:50:20.651003 1489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:33115 every 8 connection(s)
I20260928 14:50:20.651324 1359 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:20.652491 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.652074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:54511)
0928 14:50:20.652261 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.652350 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
W20260928 14:50:20.652954 1490 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:43163 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.190:43163: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
I20260928 14:50:20.653329 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.653186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:45341)
0928 14:50:20.653267 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.653300 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 14:50:20.653579 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.653521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:45845)
0928 14:50:20.653549 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.653567 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 14:50:20.653692 1490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:50:20.653911 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 1359
I20260928 14:50:20.654191 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:46743
--local_ip_for_outbound_sockets=127.18.96.131
--tserver_master_addrs=127.18.96.190:43163
--webserver_port=36341
--webserver_interface=127.18.96.131
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:20.737795 1493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:20.738021 1493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:20.738046 1493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:20.739240 1493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:20.739281 1493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:20.739308 1493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:50:20.740782 1493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:46743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=36341
--tserver_master_addrs=127.18.96.190:43163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:20.740988 1493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:20.741168 1493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:20.746291 1499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:20.746407 1501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:20.746345 1498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:20.746544 1493 server_base.cc:1034] running on GCE node
I20260928 14:50:20.746783 1493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:20.747025 1493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:20.748171 1493 hybrid_clock.cc:648] HybridClock initialized: now 1790607020748148 us; error 38 us; skew 500 ppm
Sep 28 14:50:20 dist-test-slave-j968 krb5kdc[32508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607020, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:20.750833 1493 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:50:20.751276 1493 webserver.cc:460] Webserver started at http://127.18.96.131:36341/ using document root <none> and password file <none>
I20260928 14:50:20.751472 1493 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:20.751521 1493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:20.752938 1493 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:20.753561 1508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:20.753749 1493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:20.753840 1493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "7997523c12664e30a31e70902792f240"
format_stamp: "Formatted at 2026-09-28 14:50:19 on dist-test-slave-j968"
server_key: "cfe741fcb5059e4be27d6732558227c9"
server_key_iv: "0fc88a765341e6f7f5642c4f73ac95a7"
server_key_version: "encryptionkey@0"
I20260928 14:50:20.754220 1493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:20.762362 1493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:20.764386 1493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:20.764542 1493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:20.764768 1493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:20.765074 1493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:20.765125 1493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:20.765167 1493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:20.765192 1493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:20.771298 1493 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:46743
I20260928 14:50:20.771351 1622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:46743 every 8 connection(s)
I20260928 14:50:20.771780 1493 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:20.773123 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.772532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:53281)
0928 14:50:20.772697 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.772774 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260928 14:50:20.773366 1623 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:43163 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.190:43163: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
I20260928 14:50:20.773679 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.773585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:56105)
0928 14:50:20.773634 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.773661 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 14:50:20.773928 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.773880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:48665)
0928 14:50:20.773899 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:20.773917 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 14:50:20.774115 1623 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:50:20.779389 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 1493
I20260928 14:50:21.585127 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:21.584871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:58555)
0928 14:50:21.585009 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:21.585050 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260928 14:50:21.654277 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:21.654075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:50301)
0928 14:50:21.654193 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:21.654230 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260928 14:50:21.774667 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:21.774433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:60699)
0928 14:50:21.774574 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:21.774612 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260928 14:50:22.405421 1630 raft_consensus.cc:482] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:22.405596 1630 raft_consensus.cc:504] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:22.405925 1630 leader_election.cc:288] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115), 7997523c12664e30a31e70902792f240 (127.18.96.131:46743)
Sep 28 14:50:22 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607020, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.130@KRBTEST.COM
Sep 28 14:50:22 dist-test-slave-j968 krb5kdc[32508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607020, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.131@KRBTEST.COM
I20260928 14:50:22.416603 1632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:22.406251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:36905 (local address 127.18.96.130:33115)
0928 14:50:22.406402 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:22.406415 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:22.406501 (+ 86us) server_negotiation.cc:407] Connection header received
0928 14:50:22.407109 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:22.407113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:22.407175 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:22.407346 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:22.408185 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:22.409022 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:22.410009 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:22.410412 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:22.410528 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:22.412878 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:22.412882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:22.412882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:22.412892 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:22.414874 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:22.415315 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:22.415318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:22.415318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:22.415409 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:22.415682 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:22.415683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:22.415684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:22.415982 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:22.416073 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:22.416351 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 14:50:22.416474 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":53,"threads_started":1}
I20260928 14:50:22.416620 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:22.406452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:53003 (local address 127.18.96.131:46743)
0928 14:50:22.406623 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:22.406632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:22.406650 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:50:22.407246 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:22.407249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:22.407290 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:22.407421 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:22.408134 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:22.408984 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:22.410017 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:22.410421 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:22.410538 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:22.413283 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:22.413287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:22.413287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:22.413296 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:22.415127 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:22.415465 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:22.415468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:22.415468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:22.415557 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:22.415760 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:22.415761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:22.415761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:22.416059 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:22.416148 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:22.416360 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 14:50:22.416485 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260928 14:50:22.416584 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:22.406154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:33115 (local address 127.18.96.129:36905)
0928 14:50:22.406402 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:22.406415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:22.406487 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:22.407386 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:22.407389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:22.407762 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:22.408020 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:22.408026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:22.409122 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:22.409123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:22.409916 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:22.409918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:22.410515 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:22.410516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:22.410642 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:22.411293 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:22.411297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:22.412777 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:22.414953 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:22.414960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:22.414961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:22.415207 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:22.415480 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:22.415481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:22.415481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:22.415603 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:22.416053 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:22.416059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:22.416190 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 14:50:22.416298 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:50:22.416489 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":56,"thread_start_us":62,"threads_started":1}
I20260928 14:50:22.416857 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:22.406330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:46743 (local address 127.18.96.129:53003)
0928 14:50:22.406428 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:22.406438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:22.406485 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:22.407418 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:22.407420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:22.407762 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:22.408020 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:22.408026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:22.409111 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:22.409112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:22.409916 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:22.409919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:22.410514 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:22.410515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:22.410643 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:22.411230 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:22.411238 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:22.413171 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:22.415205 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:22.415206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:22.415206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:22.415369 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:22.415627 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:22.415628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:22.415628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:22.415695 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:22.416132 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:22.416133 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:22.416196 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 14:50:22.416298 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:50:22.416577 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":164,"thread_start_us":55,"threads_started":1}
W20260928 14:50:22.417588 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
W20260928 14:50:22.417671 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7997523c12664e30a31e70902792f240 (127.18.96.131:46743): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
I20260928 14:50:22.417721 1242 leader_election.cc:302] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3400aa1331c42da855345c407bf5eba; no voters: 5fa991f12e5943eeb335769d1e36c2db, 7997523c12664e30a31e70902792f240
I20260928 14:50:22.417820 1630 raft_consensus.cc:2726] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:50:22.585839 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:22.585667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:43425)
0928 14:50:22.585746 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:22.585795 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:50:22.654989 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:22.654762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:38165)
0928 14:50:22.654894 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:22.654941 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260928 14:50:22.775580 1636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:22.775283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:56545)
0928 14:50:22.775471 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:22.775530 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260928 14:50:23.586632 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:23.586363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:53023)
0928 14:50:23.586530 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:23.586580 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260928 14:50:23.655692 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:23.655445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:52527)
0928 14:50:23.655607 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:23.655645 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260928 14:50:23.776427 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:23.776205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:38279)
0928 14:50:23.776338 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:23.776381 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260928 14:50:24.013373 1640 raft_consensus.cc:482] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:24.013479 1640 raft_consensus.cc:504] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:24.013693 1640 leader_election.cc:288] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115), 7997523c12664e30a31e70902792f240 (127.18.96.131:46743)
W20260928 14:50:24.014350 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7997523c12664e30a31e70902792f240 (127.18.96.131:46743): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
W20260928 14:50:24.014436 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
I20260928 14:50:24.014453 1242 leader_election.cc:302] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3400aa1331c42da855345c407bf5eba; no voters: 5fa991f12e5943eeb335769d1e36c2db, 7997523c12664e30a31e70902792f240
I20260928 14:50:24.014535 1640 raft_consensus.cc:2726] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:50:24.587461 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:24.587237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:34857)
0928 14:50:24.587373 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:24.587414 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260928 14:50:24.656327 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:24.656124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:54833)
0928 14:50:24.656244 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:24.656283 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260928 14:50:24.777171 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:24.776941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:48745)
0928 14:50:24.777082 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:24.777125 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260928 14:50:25.588305 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:25.588042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:39773)
0928 14:50:25.588198 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:25.588233 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260928 14:50:25.657150 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:25.656834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:60945)
0928 14:50:25.656975 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:25.657089 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260928 14:50:25.777952 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:25.777713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:35521)
0928 14:50:25.777854 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:25.777905 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260928 14:50:26.171356 1647 raft_consensus.cc:482] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:26.171474 1647 raft_consensus.cc:504] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:26.171636 1647 leader_election.cc:288] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115), 7997523c12664e30a31e70902792f240 (127.18.96.131:46743)
W20260928 14:50:26.172116 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
W20260928 14:50:26.172200 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7997523c12664e30a31e70902792f240 (127.18.96.131:46743): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
I20260928 14:50:26.172232 1242 leader_election.cc:302] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3400aa1331c42da855345c407bf5eba; no voters: 5fa991f12e5943eeb335769d1e36c2db, 7997523c12664e30a31e70902792f240
I20260928 14:50:26.172338 1647 raft_consensus.cc:2726] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:50:26.589148 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:26.588865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:42317)
0928 14:50:26.589046 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:26.589095 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260928 14:50:26.657829 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:26.657625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:52765)
0928 14:50:26.657743 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:26.657779 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260928 14:50:26.778707 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:26.778453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:58107)
0928 14:50:26.778610 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:26.778657 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260928 14:50:27.589918 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:27.589656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:33077)
0928 14:50:27.589825 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:27.589867 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260928 14:50:27.658551 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:27.658328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:37293)
0928 14:50:27.658466 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:27.658508 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260928 14:50:27.779495 1653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:27.779227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:50025)
0928 14:50:27.779400 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:27.779443 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260928 14:50:28.590777 1654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:28.590509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:56327)
0928 14:50:28.590681 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:28.590728 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260928 14:50:28.659240 1655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:28.658996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:41809)
0928 14:50:28.659145 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:28.659185 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260928 14:50:28.780364 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:28.780141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:45363)
0928 14:50:28.780275 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:28.780317 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260928 14:50:29.591857 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:29.591575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:43725)
0928 14:50:29.591769 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:29.591806 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":121,"threads_started":1}
I20260928 14:50:29.660073 1658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:29.659748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:54129)
0928 14:50:29.659951 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:29.660014 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
I20260928 14:50:29.781086 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:29.780857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:33743)
0928 14:50:29.780994 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:29.781039 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260928 14:50:30.592832 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:30.592573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:57427)
0928 14:50:30.592734 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:30.592779 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260928 14:50:30.660750 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:30.660539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:53829)
0928 14:50:30.660662 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:30.660703 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 14:50:30.781795 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:30.781543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:33821)
0928 14:50:30.781705 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:30.781746 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260928 14:50:31.073458 1663 raft_consensus.cc:482] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:31.073563 1663 raft_consensus.cc:504] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:31.073734 1663 leader_election.cc:288] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115), 7997523c12664e30a31e70902792f240 (127.18.96.131:46743)
W20260928 14:50:31.074246 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
W20260928 14:50:31.074327 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7997523c12664e30a31e70902792f240 (127.18.96.131:46743): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
I20260928 14:50:31.074345 1242 leader_election.cc:302] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3400aa1331c42da855345c407bf5eba; no voters: 5fa991f12e5943eeb335769d1e36c2db, 7997523c12664e30a31e70902792f240
I20260928 14:50:31.074424 1663 raft_consensus.cc:2726] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:50:31.593743 1664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:31.593477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:33243)
0928 14:50:31.593640 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:31.593691 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260928 14:50:31.661542 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:31.661303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:59413)
0928 14:50:31.661449 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:31.661496 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260928 14:50:31.782608 1666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:31.782363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:41579)
0928 14:50:31.782514 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:31.782560 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260928 14:50:32.594936 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:32.594583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:46441)
0928 14:50:32.594786 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:32.594867 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260928 14:50:32.662490 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:32.662213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:45877)
0928 14:50:32.662392 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:32.662442 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260928 14:50:32.737668 1669 raft_consensus.cc:482] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:32.737797 1669 raft_consensus.cc:504] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa991f12e5943eeb335769d1e36c2db" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 33115 } } peers { permanent_uuid: "7997523c12664e30a31e70902792f240" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46743 } } peers { permanent_uuid: "d3400aa1331c42da855345c407bf5eba" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 36059 } }
I20260928 14:50:32.738111 1669 leader_election.cc:288] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115), 7997523c12664e30a31e70902792f240 (127.18.96.131:46743)
W20260928 14:50:32.738557 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5fa991f12e5943eeb335769d1e36c2db (127.18.96.130:33115): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
W20260928 14:50:32.738654 1242 leader_election.cc:341] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7997523c12664e30a31e70902792f240 (127.18.96.131:46743): Not found: Tablet not found: ba85c8569f3a44f2aa0cc93a34cd5abe
I20260928 14:50:32.738689 1242 leader_election.cc:302] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3400aa1331c42da855345c407bf5eba; no voters: 5fa991f12e5943eeb335769d1e36c2db, 7997523c12664e30a31e70902792f240
I20260928 14:50:32.738787 1669 raft_consensus.cc:2726] T ba85c8569f3a44f2aa0cc93a34cd5abe P d3400aa1331c42da855345c407bf5eba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:50:32.783687 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:32.783358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:37569)
0928 14:50:32.783585 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:32.783637 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260928 14:50:33.595844 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:33.595566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:51357)
0928 14:50:33.595723 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:33.595770 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260928 14:50:33.663364 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:33.663092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:46491)
0928 14:50:33.663266 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:33.663317 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260928 14:50:33.784592 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:33.784328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:53301)
0928 14:50:33.784498 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:33.784537 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260928 14:50:34.596901 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:34.596590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:52485)
0928 14:50:34.596811 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:34.596852 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260928 14:50:34.664183 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:34.663939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:34593)
0928 14:50:34.664090 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:34.664134 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260928 14:50:34.785482 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:34.785237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:53241)
0928 14:50:34.785383 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:34.785431 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260928 14:50:35.597874 1677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:35.597605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.129:46523)
0928 14:50:35.597770 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:35.597816 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260928 14:50:35.665024 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:35.664753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.130:39595)
0928 14:50:35.664929 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:35.664971 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260928 14:50:35.780813 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 1194
I20260928 14:50:35.786123 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 1359
I20260928 14:50:35.786849 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:35.786218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43163 (local address 127.18.96.131:43013)
0928 14:50:35.786748 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:35.786787 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:43163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":77,"threads_started":1}
I20260928 14:50:35.790808 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 1493
2026-09-28T14:50:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18557 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:35 dist-test-slave-j968 krb5kdc[1682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:35 dist-test-slave-j968 krb5kdc[1682](info): set up 2 sockets
Sep 28 14:50:35 dist-test-slave-j968 krb5kdc[1682](info): commencing operation
krb5kdc: starting...
W20260928 14:50:36.865108 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:50:36 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:50:36.999629 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:46593
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:37.083693 1695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:37.083957 1695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:37.083993 1695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:37.085139 1695 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:50:37.085172 1695 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:50:37.085186 1695 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:50:37.085197 1695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:37.085210 1695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:37.085223 1695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:37.085235 1695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:37.085247 1695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:37.086726 1695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:46593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:37.086942 1695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:37.087131 1695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:37.092684 1701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:37.092804 1695 server_base.cc:1034] running on GCE node
W20260928 14:50:37.092736 1703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:37.092686 1700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:37.093344 1695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:37.093693 1695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:37.094890 1695 hybrid_clock.cc:648] HybridClock initialized: now 1790607037094865 us; error 48 us; skew 500 ppm
Sep 28 14:50:37 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:37.105491 1695 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:50:37.106055 1695 webserver.cc:460] Webserver started at http://127.18.96.190:43553/ using document root <none> and password file <none>
I20260928 14:50:37.106325 1695 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:37.106400 1695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:37.106609 1695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:37.107774 1695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "9600b5ccece148c9b7d72203719e8ad6"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "4941c8fb0876a176bc88046bc0386b73"
server_key_iv: "21cd08e92b55a72fbeba391e1297ce8e"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.108227 1695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "9600b5ccece148c9b7d72203719e8ad6"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "4941c8fb0876a176bc88046bc0386b73"
server_key_iv: "21cd08e92b55a72fbeba391e1297ce8e"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.109730 1695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:37.110642 1710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.110888 1695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:37.111007 1695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "9600b5ccece148c9b7d72203719e8ad6"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "4941c8fb0876a176bc88046bc0386b73"
server_key_iv: "21cd08e92b55a72fbeba391e1297ce8e"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.111117 1695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:37.131472 1695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:37.133920 1695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:37.134115 1695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:37.138489 1695 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:46593
I20260928 14:50:37.138545 1763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:46593 every 8 connection(s)
I20260928 14:50:37.138962 1695 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:37.139994 1764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:37.142356 1764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Bootstrap starting.
I20260928 14:50:37.142932 1764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:37.143179 1764 log.cc:824] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:37.143932 1764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: No bootstrap required, opened a new log
I20260928 14:50:37.145363 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 1695
I20260928 14:50:37.145489 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:50:37.145699 18818 external_mini_cluster.cc:1428] Setting key 636be2d1225c8b5c96a22e41ea124159
I20260928 14:50:37.145735 1764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:50:37.145951 1764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:37.145998 1764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9600b5ccece148c9b7d72203719e8ad6, State: Initialized, Role: FOLLOWER
I20260928 14:50:37.146507 1764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:50:37.146621 1764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:37.146658 1764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:37.146719 1764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:37.147977 1764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:50:37.148105 1764 leader_election.cc:302] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9600b5ccece148c9b7d72203719e8ad6; no voters:
I20260928 14:50:37.148303 1764 leader_election.cc:288] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:37.148361 1769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:37.148501 1769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 9600b5ccece148c9b7d72203719e8ad6, State: Running, Role: LEADER
I20260928 14:50:37.148589 1764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:37.148651 1769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:50:37.148985 1769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9600b5ccece148c9b7d72203719e8ad6. Latest consensus state: current_term: 1 leader_uuid: "9600b5ccece148c9b7d72203719e8ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } } }
I20260928 14:50:37.149049 1769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:37.149008 1770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9600b5ccece148c9b7d72203719e8ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } } }
I20260928 14:50:37.149291 1770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:37.149883 1775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:37.150487 1775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:37.152189 1775 catalog_manager.cc:1347] Generated new cluster ID: f07a4da4b44e44a98486401de377404a
I20260928 14:50:37.152247 1775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:50:37 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:37.158525 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.146281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.18.96.190:46593)
0928 14:50:37.146470 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.146479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.146508 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:50:37.146611 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.146617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.146692 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.146841 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:37.147688 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.149088 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.150223 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.150831 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.150951 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.154937 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:37.154941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:37.154942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:37.154957 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:37.156790 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.157210 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.157213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.157214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.157334 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.157565 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.157566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.157566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.157846 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:37.158019 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.158131 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.158292 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260928 14:50:37.161314 1775 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:37.161857 1775 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:37.165005 1775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Generated new TSK 0
I20260928 14:50:37.165205 1775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:50:37.198910 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:46593
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:37.284461 1790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:37.284684 1790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:37.284710 1790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:37.285873 1790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:37.285912 1790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:37.285938 1790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:50:37.287417 1790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:37.287633 1790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:37.287833 1790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:37.293584 1795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:37.293758 1796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:37.293754 1798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:37.294888 1790 server_base.cc:1034] running on GCE node
I20260928 14:50:37.295312 1790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:37.295567 1790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:37.296743 1790 hybrid_clock.cc:648] HybridClock initialized: now 1790607037296659 us; error 105 us; skew 500 ppm
Sep 28 14:50:37 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:37.299773 1790 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:50:37.300340 1790 webserver.cc:460] Webserver started at http://127.18.96.129:33605/ using document root <none> and password file <none>
I20260928 14:50:37.300647 1790 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:37.300722 1790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:37.300884 1790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:37.302136 1790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "bb1b416518bf4bc2898e5cb618b99fbe"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "661636a53aa3eb86dc43e0c7ce17d14d"
server_key_iv: "7c82e3d068a3fc26d342478814202f92"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.302517 1790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "bb1b416518bf4bc2898e5cb618b99fbe"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "661636a53aa3eb86dc43e0c7ce17d14d"
server_key_iv: "7c82e3d068a3fc26d342478814202f92"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.304026 1790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:50:37.304963 1805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.305292 1790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:37.305418 1790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "bb1b416518bf4bc2898e5cb618b99fbe"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "661636a53aa3eb86dc43e0c7ce17d14d"
server_key_iv: "7c82e3d068a3fc26d342478814202f92"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.305524 1790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:37.319339 1790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:37.322364 1790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:37.322585 1790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:37.322858 1790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:37.323228 1790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:37.323288 1790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.323333 1790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:37.323361 1790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.329789 1790 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:38457
I20260928 14:50:37.329833 1919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:38457 every 8 connection(s)
I20260928 14:50:37.330211 1790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:37.334712 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 1790
I20260928 14:50:37.334882 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:50:37.335059 18818 external_mini_cluster.cc:1428] Setting key 4c3c1c8f1089c1acf669caede43dfb67
Sep 28 14:50:37 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:37.341537 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.331067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:45451 (local address 127.18.96.190:46593)
0928 14:50:37.331156 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.331158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.331309 (+ 151us) server_negotiation.cc:407] Connection header received
0928 14:50:37.331836 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.331837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.331855 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.331957 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:37.333118 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.333707 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.334881 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.335184 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.335252 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.337963 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:37.337965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:37.337965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:37.337979 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:37.339721 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.340363 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.340364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.340365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.340432 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.340793 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.340794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.340795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.340921 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:37.340978 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.341385 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.341468 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:50:37.341923 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.330970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:45451)
0928 14:50:37.331153 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:37.331198 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:37.331712 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:37.332094 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:37.332098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:37.332764 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:37.332982 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.332988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.333810 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.333811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.334754 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.334757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.335263 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.335264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.335409 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.336081 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:37.336093 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:37.337852 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:37.339825 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:37.339839 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:37.339843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:37.340254 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:37.340517 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:37.340518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:37.340518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:37.340713 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:37.340994 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:37.341002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:37.341221 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 14:50:37.341364 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 14:50:37.341592 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260928 14:50:37.342527 1920 heartbeater.cc:347] Connected to a master server at 127.18.96.190:46593
I20260928 14:50:37.342666 1920 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:37.342897 1920 heartbeater.cc:511] Master 127.18.96.190:46593 requested a full tablet report, sending...
I20260928 14:50:37.343508 1727 ts_manager.cc:194] Registered new tserver with Master: bb1b416518bf4bc2898e5cb618b99fbe (127.18.96.129:38457)
I20260928 14:50:37.344224 1727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:45451
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:50:37.367183 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:46593
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:37.458915 1925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:37.459172 1925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:37.459201 1925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:37.460625 1925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:37.460677 1925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:37.460711 1925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:50:37.462433 1925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:37.462728 1925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:37.462989 1925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:37.468679 1931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:37.468717 1925 server_base.cc:1034] running on GCE node
W20260928 14:50:37.468614 1933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:37.468622 1930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:37.469025 1925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:37.469323 1925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:37.470501 1925 hybrid_clock.cc:648] HybridClock initialized: now 1790607037470502 us; error 46 us; skew 500 ppm
Sep 28 14:50:37 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:37.473388 1925 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:50:37.473959 1925 webserver.cc:460] Webserver started at http://127.18.96.130:32999/ using document root <none> and password file <none>
I20260928 14:50:37.474215 1925 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:37.474290 1925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:37.474450 1925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:37.475646 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "6dd630eee3584b48be15995727ebe03c"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "c0783437c5e14ea47b7a4b0dac949356"
server_key_iv: "8d8d307a72968dd5e20aa87d8479c307"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.476092 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "6dd630eee3584b48be15995727ebe03c"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "c0783437c5e14ea47b7a4b0dac949356"
server_key_iv: "8d8d307a72968dd5e20aa87d8479c307"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.477691 1925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:37.478484 1940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.478758 1925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:37.478888 1925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "6dd630eee3584b48be15995727ebe03c"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "c0783437c5e14ea47b7a4b0dac949356"
server_key_iv: "8d8d307a72968dd5e20aa87d8479c307"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.478966 1925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:37.503363 1925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:37.505880 1925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:37.506080 1925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:37.506359 1925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:37.506709 1925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:37.506769 1925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.506825 1925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:37.506851 1925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.513664 1925 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:41927
I20260928 14:50:37.513710 2054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:41927 every 8 connection(s)
I20260928 14:50:37.514086 1925 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:50:37 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:37.523042 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 1925
I20260928 14:50:37.523186 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:50:37.523375 18818 external_mini_cluster.cc:1428] Setting key ea521e1defcb648e5150612786beb97c
I20260928 14:50:37.524755 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.514873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:54141)
0928 14:50:37.515123 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:37.515164 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:37.515712 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:37.516018 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:37.516021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:37.516664 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:37.516881 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.516886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.517705 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.517706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.518527 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.518530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.518979 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.518980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.519108 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.519680 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:37.519688 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:37.521284 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:37.523125 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:37.523135 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:37.523137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:37.523430 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:37.523710 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:37.523711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:37.523711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:37.523857 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:37.524182 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:37.524189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:37.524346 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 14:50:37.524433 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:50:37.524576 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":108,"threads_started":1}
I20260928 14:50:37.524763 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.514898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:54141 (local address 127.18.96.190:46593)
0928 14:50:37.514983 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.514985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.515260 (+ 275us) server_negotiation.cc:407] Connection header received
0928 14:50:37.515828 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.515829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.515843 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.515919 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:37.516996 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.517598 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.518637 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.518853 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.518926 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.521399 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:37.521400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:37.521401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:37.521409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:37.523011 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.523534 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.523537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.523537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.523617 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.523937 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.523938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.523938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.524106 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:37.524180 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.524485 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.524586 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:50:37.525256 2055 heartbeater.cc:347] Connected to a master server at 127.18.96.190:46593
I20260928 14:50:37.525362 2055 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:37.525548 2055 heartbeater.cc:511] Master 127.18.96.190:46593 requested a full tablet report, sending...
I20260928 14:50:37.526012 1727 ts_manager.cc:194] Registered new tserver with Master: 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927)
I20260928 14:50:37.526549 1727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:54141
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:50:37.554783 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:46593
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:37.640715 2060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:37.641000 2060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:37.641042 2060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:37.642323 2060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:37.642412 2060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:37.642462 2060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:50:37.644045 2060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:37.644338 2060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:37.644586 2060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:37.650364 2069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:37.650390 2066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:37.650355 2065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:37.650727 2060 server_base.cc:1034] running on GCE node
I20260928 14:50:37.650919 2060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:37.651124 2060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:37.652284 2060 hybrid_clock.cc:648] HybridClock initialized: now 1790607037652282 us; error 35 us; skew 500 ppm
Sep 28 14:50:37 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:37.654759 2060 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:50:37.655282 2060 webserver.cc:460] Webserver started at http://127.18.96.131:34447/ using document root <none> and password file <none>
I20260928 14:50:37.655510 2060 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:37.655577 2060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:37.655736 2060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:37.656879 2060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "3786aa5155a34d32a1489c4ab695d12a"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "b0c9d7b7a3f2aafac34572315531c057"
server_key_iv: "feaf337565ace9f5a12b0b41e5bb7f8a"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.657261 2060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "3786aa5155a34d32a1489c4ab695d12a"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "b0c9d7b7a3f2aafac34572315531c057"
server_key_iv: "feaf337565ace9f5a12b0b41e5bb7f8a"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.658651 2060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:50:37.659487 2076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.659696 2060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:37.659799 2060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "3786aa5155a34d32a1489c4ab695d12a"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "b0c9d7b7a3f2aafac34572315531c057"
server_key_iv: "feaf337565ace9f5a12b0b41e5bb7f8a"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.659931 2060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:37.668879 2060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:37.671069 2060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:37.671262 2060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:37.671523 2060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:37.671924 2060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:37.671993 2060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.672039 2060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:37.672068 2060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.678836 2060 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:34879
I20260928 14:50:37.678918 2189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:34879 every 8 connection(s)
I20260928 14:50:37.679299 2060 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:37.680361 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 2060
I20260928 14:50:37.680449 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:50:37.680624 18818 external_mini_cluster.cc:1428] Setting key 9ae3fd9d89d880d0e96f581b7f1bea7d
Sep 28 14:50:37 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607037, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:37.689766 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.680141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:59819 (local address 127.18.96.190:46593)
0928 14:50:37.680188 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.680191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.680434 (+ 243us) server_negotiation.cc:407] Connection header received
0928 14:50:37.681229 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.681230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.681246 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.681294 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:37.682454 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.683016 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.684244 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.684475 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.684569 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.686805 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:37.686807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:37.686807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:37.686816 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:37.688187 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.688680 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.688681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.688682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.688739 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.689017 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.689018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.689019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.689157 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:37.689241 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.689562 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.689668 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:50:37.689958 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.680082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:59819)
0928 14:50:37.680285 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:37.680327 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:37.681061 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:37.681391 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:37.681395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:37.682060 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:37.682330 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.682335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.683118 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.683119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.684133 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.684136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.684581 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.684582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.684725 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.685245 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:37.685255 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:37.686697 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:37.688292 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:37.688300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:37.688303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:37.688543 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:37.688816 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:37.688817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:37.688818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:37.688935 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:37.689245 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:37.689253 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:37.689402 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 14:50:37.689517 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:50:37.689695 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":120,"threads_started":1}
I20260928 14:50:37.690538 2190 heartbeater.cc:347] Connected to a master server at 127.18.96.190:46593
I20260928 14:50:37.690644 2190 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:37.690871 2190 heartbeater.cc:511] Master 127.18.96.190:46593 requested a full tablet report, sending...
I20260928 14:50:37.691322 1727 ts_manager.cc:194] Registered new tserver with Master: 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
I20260928 14:50:37.691756 1727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:59819
I20260928 14:50:37.692690 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:37.701140 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.694388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.18.96.190:46593)
0928 14:50:37.694462 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.694464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.694643 (+ 179us) server_negotiation.cc:407] Connection header received
0928 14:50:37.694684 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.694685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.694694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.694739 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:37.695371 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.695980 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.696748 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.697014 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.697089 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.698505 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:37.698507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:37.698507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:37.698515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:37.699953 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.700278 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.700279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.700279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.700337 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.700515 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.700516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.700516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.700695 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:37.700778 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.700897 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.700974 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:50:37.702940 1727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:37.704667 1727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:37.712670 2207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.708218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.18.96.130:41927)
0928 14:50:37.708772 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.708775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.708783 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:50:37.708812 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.708813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.708824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.708881 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:37.709070 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.710130 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.711441 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.712058 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.712267 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.712349 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.712526 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.712575 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":97,"threads_started":1}
I20260928 14:50:37.712652 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.707875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:41927 (local address 127.0.0.1:43086)
0928 14:50:37.708223 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:37.708233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:37.708283 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:37.708910 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:37.708911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:37.708913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:37.708966 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.708976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.710247 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.710248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.711382 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.711383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.712268 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.712270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.712369 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.712383 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:50:37.712461 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:50:37.712518 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260928 14:50:37.712675 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.707579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38457 (local address 127.0.0.1:51046)
0928 14:50:37.707924 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:37.707935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:37.708000 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:37.708493 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:37.708496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:37.708498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:37.708619 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.708620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.709841 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.709842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.711268 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.711272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.712298 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.712300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.712389 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.712393 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:50:37.712437 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:50:37.712539 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":78,"thread_start_us":173,"threads_started":1}
I20260928 14:50:37.713055 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.708080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.18.96.129:38457)
0928 14:50:37.708318 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.708325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.708336 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:50:37.708381 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.708383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.708400 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.708491 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:37.708681 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.709727 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.711376 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.712195 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.712699 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.712860 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.712891 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.712959 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260928 14:50:37.713199 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.707712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.18.96.131:34879)
0928 14:50:37.707917 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.707923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.707973 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:50:37.708404 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.708408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.708427 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.708504 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:37.708954 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.710092 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.711290 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.712163 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.712252 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.712326 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.713024 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.713082 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260928 14:50:37.713318 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.707573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:34879 (local address 127.0.0.1:38914)
0928 14:50:37.707819 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:37.707911 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:37.708001 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:37.708586 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:37.708587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:37.708588 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:37.708619 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.708620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.710216 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.710217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.711187 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.711190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.712357 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.712358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.712415 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.712420 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:50:37.712471 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:50:37.712539 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260928 14:50:37.713871 1853 tablet_service.cc:1467] Processing CreateTablet for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 (DEFAULT_TABLE table=test-table [id=fc1895dbedd647b18edcac205d34c890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:37.714270 1853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:37.714197 2124 tablet_service.cc:1467] Processing CreateTablet for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 (DEFAULT_TABLE table=test-table [id=fc1895dbedd647b18edcac205d34c890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:37.714497 2124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:37.717257 2208 tablet_bootstrap.cc:492] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: Bootstrap starting.
I20260928 14:50:37.717386 2209 tablet_bootstrap.cc:492] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Bootstrap starting.
I20260928 14:50:37.717808 1988 tablet_service.cc:1467] Processing CreateTablet for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 (DEFAULT_TABLE table=test-table [id=fc1895dbedd647b18edcac205d34c890]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:37.718079 2208 tablet_bootstrap.cc:654] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:37.718107 2209 tablet_bootstrap.cc:654] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:37.718200 1988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:37.718669 2209 log.cc:824] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:37.718670 2208 log.cc:824] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:37.719501 2209 tablet_bootstrap.cc:492] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: No bootstrap required, opened a new log
I20260928 14:50:37.719599 2209 ts_tablet_manager.cc:1397] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:50:37.719655 2208 tablet_bootstrap.cc:492] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: No bootstrap required, opened a new log
I20260928 14:50:37.719726 2208 ts_tablet_manager.cc:1397] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:50:37.721203 2209 raft_consensus.cc:348] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:37.721400 2209 raft_consensus.cc:374] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:37.721468 2209 raft_consensus.cc:729] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1b416518bf4bc2898e5cb618b99fbe, State: Initialized, Role: FOLLOWER
I20260928 14:50:37.721673 2209 consensus_queue.cc:260] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:37.721827 2208 raft_consensus.cc:348] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:37.722021 2209 ts_tablet_manager.cc:1428] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:50:37.722067 2208 raft_consensus.cc:374] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:37.722136 2212 tablet_bootstrap.cc:492] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Bootstrap starting.
I20260928 14:50:37.722183 1920 heartbeater.cc:503] Master 127.18.96.190:46593 was elected leader, sending a full tablet report...
I20260928 14:50:37.722195 2208 raft_consensus.cc:729] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3786aa5155a34d32a1489c4ab695d12a, State: Initialized, Role: FOLLOWER
I20260928 14:50:37.722324 2208 consensus_queue.cc:260] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:37.722662 2208 ts_tablet_manager.cc:1428] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:37.722815 2190 heartbeater.cc:503] Master 127.18.96.190:46593 was elected leader, sending a full tablet report...
I20260928 14:50:37.722987 2212 tablet_bootstrap.cc:654] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:37.723644 2212 log.cc:824] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:37.724468 2212 tablet_bootstrap.cc:492] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: No bootstrap required, opened a new log
I20260928 14:50:37.724529 2212 ts_tablet_manager.cc:1397] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:50:37.726033 2212 raft_consensus.cc:348] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:37.726168 2212 raft_consensus.cc:374] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:37.726190 2212 raft_consensus.cc:729] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd630eee3584b48be15995727ebe03c, State: Initialized, Role: FOLLOWER
I20260928 14:50:37.726322 2212 consensus_queue.cc:260] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:37.726619 2212 ts_tablet_manager.cc:1428] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:50:37.726701 2055 heartbeater.cc:503] Master 127.18.96.190:46593 was elected leader, sending a full tablet report...
W20260928 14:50:37.764755 2056 tablet.cc:2369] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:50:37.831089 1921 tablet.cc:2369] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:50:37.929970 2191 tablet.cc:2369] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:37.935261 2216 raft_consensus.cc:482] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:37.935389 2216 raft_consensus.cc:504] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:37.935848 2216 leader_election.cc:288] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb1b416518bf4bc2898e5cb618b99fbe (127.18.96.129:38457), 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
I20260928 14:50:37.939198 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.936165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:60877 (local address 127.18.96.131:34879)
0928 14:50:37.936221 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.936224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.936606 (+ 382us) server_negotiation.cc:407] Connection header received
0928 14:50:37.936638 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.936639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.936648 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.936703 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:37.936831 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.937331 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.938390 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.938803 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.938897 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.938906 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.939075 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.939119 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:50:37.939198 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.936170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:38519 (local address 127.18.96.129:38457)
0928 14:50:37.936244 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.936246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.936256 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:50:37.936287 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.936288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.936302 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.936377 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:37.936753 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.937300 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.938372 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.938769 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.938868 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.938879 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.939072 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.939121 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:50:37.939209 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.936094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38457 (local address 127.18.96.130:38519)
0928 14:50:37.936140 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:37.936151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:37.936223 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:37.936395 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:37.936396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:37.936398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:37.936509 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.936512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.937411 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.937412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.938264 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.938268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.938868 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.938869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.938964 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.938968 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:50:37.939035 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:50:37.939088 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":1}
I20260928 14:50:37.939224 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.936095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:34879 (local address 127.18.96.130:60877)
0928 14:50:37.936491 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:37.936501 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:37.936571 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:37.936706 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:37.936706 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:37.936707 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:37.936757 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.936758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.937431 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.937432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.938282 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:37.938285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.938907 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.938909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:37.938985 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.938999 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:50:37.939069 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:50:37.939119 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":185,"thread_start_us":133,"threads_started":1}
I20260928 14:50:37.939575 1873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082c5a8dcb1b4b248a2b4bc8c9d3b8a0" candidate_uuid: "6dd630eee3584b48be15995727ebe03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" is_pre_election: true
I20260928 14:50:37.939615 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082c5a8dcb1b4b248a2b4bc8c9d3b8a0" candidate_uuid: "6dd630eee3584b48be15995727ebe03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3786aa5155a34d32a1489c4ab695d12a" is_pre_election: true
I20260928 14:50:37.939759 2144 raft_consensus.cc:2454] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd630eee3584b48be15995727ebe03c in term 0.
I20260928 14:50:37.939759 1873 raft_consensus.cc:2454] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd630eee3584b48be15995727ebe03c in term 0.
I20260928 14:50:37.940057 1942 leader_election.cc:302] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd630eee3584b48be15995727ebe03c, bb1b416518bf4bc2898e5cb618b99fbe; no voters:
I20260928 14:50:37.940234 2216 raft_consensus.cc:2781] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:37.940330 2216 raft_consensus.cc:482] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:37.940368 2216 raft_consensus.cc:3037] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:37.941238 2216 raft_consensus.cc:504] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:37.941426 2216 leader_election.cc:288] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [CANDIDATE]: Term 1 election: Requested vote from peers bb1b416518bf4bc2898e5cb618b99fbe (127.18.96.129:38457), 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
I20260928 14:50:37.941694 1873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082c5a8dcb1b4b248a2b4bc8c9d3b8a0" candidate_uuid: "6dd630eee3584b48be15995727ebe03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1b416518bf4bc2898e5cb618b99fbe"
I20260928 14:50:37.941694 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082c5a8dcb1b4b248a2b4bc8c9d3b8a0" candidate_uuid: "6dd630eee3584b48be15995727ebe03c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3786aa5155a34d32a1489c4ab695d12a"
I20260928 14:50:37.941795 2144 raft_consensus.cc:3037] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:37.941795 1873 raft_consensus.cc:3037] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:37.942770 1873 raft_consensus.cc:2454] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd630eee3584b48be15995727ebe03c in term 1.
I20260928 14:50:37.942768 2144 raft_consensus.cc:2454] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd630eee3584b48be15995727ebe03c in term 1.
I20260928 14:50:37.942979 1941 leader_election.cc:302] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3786aa5155a34d32a1489c4ab695d12a, 6dd630eee3584b48be15995727ebe03c; no voters:
I20260928 14:50:37.943128 2216 raft_consensus.cc:2781] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:37.943333 2216 raft_consensus.cc:686] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 1 LEADER]: Becoming Leader. State: Replica: 6dd630eee3584b48be15995727ebe03c, State: Running, Role: LEADER
I20260928 14:50:37.943459 2216 consensus_queue.cc:237] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:37.944557 1727 catalog_manager.cc:5495] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd630eee3584b48be15995727ebe03c (127.18.96.130). New cstate: current_term: 1 leader_uuid: "6dd630eee3584b48be15995727ebe03c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } health_report { overall_health: HEALTHY } } }
I20260928 14:50:37.996876 2207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.993497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.18.96.130:41927)
0928 14:50:37.993575 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.993577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.993613 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:50:37.993729 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.993730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.993750 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.993868 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:37.994162 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.994650 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.996042 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.996268 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.996331 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.996520 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:37.996648 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:37.996717 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.996768 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.996789 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:50:37.998988 2144 raft_consensus.cc:1261] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [term 1 FOLLOWER]: Refusing update from remote peer 6dd630eee3584b48be15995727ebe03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:37.998988 1873 raft_consensus.cc:1261] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Refusing update from remote peer 6dd630eee3584b48be15995727ebe03c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:37.999416 2218 consensus_queue.cc:1035] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:37.999576 2216 consensus_queue.cc:1035] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:38.001916 2221 mvcc.cc:205] Tried to move back new op lower bound from 7334326427640320000 to 7334326427417141248. Current Snapshot: MvccSnapshot[applied={T|T < 7334326427640320000}]
I20260928 14:50:38.003207 2222 mvcc.cc:205] Tried to move back new op lower bound from 7334326427640320000 to 7334326427417141248. Current Snapshot: MvccSnapshot[applied={T|T < 7334326427640320000}]
I20260928 14:50:38.002913 2223 mvcc.cc:205] Tried to move back new op lower bound from 7334326427640320000 to 7334326427417141248. Current Snapshot: MvccSnapshot[applied={T|T < 7334326427640320000}]
I20260928 14:50:38.004477 1726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:38.004573 1726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:38.005437 1726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Sending DeleteTablet for 3 replicas of tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0
I20260928 14:50:38.005769 1853 tablet_service.cc:1514] Processing DeleteTablet for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:38 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:51046
I20260928 14:50:38.005787 2124 tablet_service.cc:1514] Processing DeleteTablet for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:38 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:38914
I20260928 14:50:38.005791 1988 tablet_service.cc:1514] Processing DeleteTablet for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:38 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:43086
I20260928 14:50:38.006014 2233 tablet_replica.cc:330] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: stopping tablet replica
I20260928 14:50:38.006083 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 1790
I20260928 14:50:38.006114 2233 raft_consensus.cc:2229] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 14:50:38.011653 1712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:38457 (error 108)
I20260928 14:50:38.011760 2232 tablet_replica.cc:330] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: stopping tablet replica
I20260928 14:50:38.011914 2232 raft_consensus.cc:2229] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:38.012072 2232 raft_consensus.cc:2258] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:38.012259 2231 tablet_replica.cc:330] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: stopping tablet replica
I20260928 14:50:38.012377 2231 raft_consensus.cc:2229] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:50:38.012502 2231 raft_consensus.cc:2258] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:50:38.012789 2202 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:38.012462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38457 (local address 127.0.0.1:51062)
0928 14:50:38.012532 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:38.012545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:38.012595 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:38.012733 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:38457: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:50:38.013056 2231 ts_tablet_manager.cc:1905] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:50:38.013056 1712 catalog_manager.cc:4584] TS bb1b416518bf4bc2898e5cb618b99fbe (127.18.96.129:38457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0: Network error: Client connection negotiation failed: client connection to 127.18.96.129:38457: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:50:38.013126 2232 ts_tablet_manager.cc:1905] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:38.013602 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 1925
I20260928 14:50:38.015477 2232 ts_tablet_manager.cc:1918] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:38.015563 2232 log.cc:1196] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/082c5a8dcb1b4b248a2b4bc8c9d3b8a0
I20260928 14:50:38.015920 2232 ts_tablet_manager.cc:1939] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 3786aa5155a34d32a1489c4ab695d12a: Deleting consensus metadata
I20260928 14:50:38.016851 1711 catalog_manager.cc:4841] TS 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879): tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 (table test-table [id=fc1895dbedd647b18edcac205d34c890]) successfully deleted
I20260928 14:50:38.020722 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:38.020536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:41927 (local address 127.0.0.1:43098)
0928 14:50:38.020606 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:38.020673 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.130:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 14:50:38.020702 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 2060
I20260928 14:50:38.026975 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 1695
I20260928 14:50:58.033172 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:46593
--webserver_interface=127.18.96.190
--webserver_port=43553
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:58.114984 2235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:58.115204 2235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:58.115227 2235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:58.116371 2235 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:50:58.116405 2235 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:50:58.116418 2235 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:50:58.116430 2235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:58.116442 2235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:58.116456 2235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:58.116467 2235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:58.116478 2235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:58.117909 2235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:46593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=43553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:58.118115 2235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:58.118288 2235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:58.128536 2242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:58.128564 2244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:58.128564 2241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:58.128998 2235 server_base.cc:1034] running on GCE node
I20260928 14:50:58.129330 2235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:58.129766 2235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:58.130986 2235 hybrid_clock.cc:648] HybridClock initialized: now 1790607058130970 us; error 34 us; skew 500 ppm
Sep 28 14:50:58 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607058, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:58.138844 2235 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:50:58.139528 2235 webserver.cc:460] Webserver started at http://127.18.96.190:43553/ using document root <none> and password file <none>
I20260928 14:50:58.139902 2235 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:58.139998 2235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:58.142238 2235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:50:58.143162 2251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:58.143386 2235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:58.143548 2235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "9600b5ccece148c9b7d72203719e8ad6"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "4941c8fb0876a176bc88046bc0386b73"
server_key_iv: "21cd08e92b55a72fbeba391e1297ce8e"
server_key_version: "encryptionkey@0"
I20260928 14:50:58.144088 2235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:58.162532 2235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:58.165124 2235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:58.165304 2235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:58.169770 2235 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:46593
I20260928 14:50:58.169811 2303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:46593 every 8 connection(s)
I20260928 14:50:58.170248 2235 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:58.171943 2304 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:50:58.172708 2304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Bootstrap starting.
I20260928 14:50:58.179060 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 2235
I20260928 14:50:58.179397 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:38457
--local_ip_for_outbound_sockets=127.18.96.129
--tserver_master_addrs=127.18.96.190:46593
--webserver_port=33605
--webserver_interface=127.18.96.129
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:58.182890 2304 log.cc:824] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:58.186108 2304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:50:58.186515 2304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Bootstrap complete.
I20260928 14:50:58.191121 2304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:50:58.191635 2304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9600b5ccece148c9b7d72203719e8ad6, State: Initialized, Role: FOLLOWER
I20260928 14:50:58.191846 2304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:50:58.191977 2304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:58.192050 2304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:58.192119 2304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:50:58.193694 2304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:50:58.193877 2304 leader_election.cc:302] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9600b5ccece148c9b7d72203719e8ad6; no voters:
I20260928 14:50:58.194171 2304 leader_election.cc:288] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:50:58.194206 2308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:50:58.194339 2308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 2 LEADER]: Becoming Leader. State: Replica: 9600b5ccece148c9b7d72203719e8ad6, State: Running, Role: LEADER
I20260928 14:50:58.194475 2304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:58.194561 2308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:50:58.194887 2308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9600b5ccece148c9b7d72203719e8ad6. Latest consensus state: current_term: 2 leader_uuid: "9600b5ccece148c9b7d72203719e8ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } } }
I20260928 14:50:58.195008 2308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:58.195194 2315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:58.194885 2309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9600b5ccece148c9b7d72203719e8ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } } }
I20260928 14:50:58.195327 2309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:58.196120 2315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:58.196226 2315 catalog_manager.cc:1259] Loaded cluster ID: f07a4da4b44e44a98486401de377404a
I20260928 14:50:58.196260 2315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:58.197432 2315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:58.197636 2315 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Loaded TSK: 0
I20260928 14:50:58.207978 2315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Generated new TSK 1
I20260928 14:50:58.208102 2315 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:50:58.275187 2306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:58.275444 2306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:58.275487 2306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:58.276839 2306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:58.276917 2306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:58.276973 2306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:50:58.278555 2306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:38457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=33605
--tserver_master_addrs=127.18.96.190:46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:58.278841 2306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:58.279106 2306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:58.284621 2334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:58.285043 2306 server_base.cc:1034] running on GCE node
W20260928 14:50:58.284583 2331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:58.284588 2332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:58.285470 2306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:58.285815 2306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:58.287014 2306 hybrid_clock.cc:648] HybridClock initialized: now 1790607058286997 us; error 33 us; skew 500 ppm
Sep 28 14:50:58 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607058, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:58.290014 2306 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:50:58.290563 2306 webserver.cc:460] Webserver started at http://127.18.96.129:33605/ using document root <none> and password file <none>
I20260928 14:50:58.290841 2306 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:58.290943 2306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:58.292723 2306 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:50:58.293651 2341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:58.293952 2306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:58.294174 2306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "bb1b416518bf4bc2898e5cb618b99fbe"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "661636a53aa3eb86dc43e0c7ce17d14d"
server_key_iv: "7c82e3d068a3fc26d342478814202f92"
server_key_version: "encryptionkey@0"
I20260928 14:50:58.294643 2306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:58.312215 2306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:58.314765 2306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:58.314947 2306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:58.315210 2306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:58.315671 2348 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:50:58.317657 2306 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:50:58.317756 2306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 14:50:58.317806 2306 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:50:58.318511 2306 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:50:58.318605 2306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 14:50:58.318645 2348 tablet_bootstrap.cc:492] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Bootstrap starting.
I20260928 14:50:58.327138 2306 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:38457
I20260928 14:50:58.327216 2456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:38457 every 8 connection(s)
I20260928 14:50:58.327911 2306 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:58.331305 2348 log.cc:824] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:58.334790 2348 tablet_bootstrap.cc:492] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:50:58.335153 2348 tablet_bootstrap.cc:492] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Bootstrap complete.
I20260928 14:50:58.335908 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 2306
I20260928 14:50:58.336220 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:41927
--local_ip_for_outbound_sockets=127.18.96.130
--tserver_master_addrs=127.18.96.190:46593
--webserver_port=32999
--webserver_interface=127.18.96.130
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:58.337271 2348 ts_tablet_manager.cc:1397] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
Sep 28 14:50:58 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607058, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:58.338845 2348 raft_consensus.cc:348] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:58.339260 2348 raft_consensus.cc:729] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1b416518bf4bc2898e5cb618b99fbe, State: Initialized, Role: FOLLOWER
I20260928 14:50:58.339439 2348 consensus_queue.cc:260] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } }
I20260928 14:50:58.339835 2348 ts_tablet_manager.cc:1428] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 14:50:58.342605 2460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.329528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:47913 (local address 127.18.96.190:46593)
0928 14:50:58.329802 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.329814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.329864 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:50:58.330674 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.330683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.330774 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.330954 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:58.331948 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.332846 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.334565 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.335197 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.335316 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.338145 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:58.338151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:58.338151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:58.338166 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:58.340475 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.340928 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.340932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.340932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.341075 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.341355 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.341356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.341357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.341707 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:58.341802 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.342238 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.342379 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260928 14:50:58.342599 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.329087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:47913)
0928 14:50:58.329316 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:58.329371 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:58.330351 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:58.331022 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:58.331027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:58.331453 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:58.331755 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.331763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.332979 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.332980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.334434 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.334437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.335434 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.335436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.335606 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.336249 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:58.336265 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:58.337982 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:58.340564 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:58.340571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:58.340573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:58.340817 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:58.341157 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:58.341158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:58.341158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:58.341278 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:58.341799 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:58.341806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:58.341967 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 14:50:58.342172 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 14:50:58.342379 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":118,"threads_started":1}
I20260928 14:50:58.343443 2457 heartbeater.cc:347] Connected to a master server at 127.18.96.190:46593
I20260928 14:50:58.343547 2457 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:58.343765 2457 heartbeater.cc:511] Master 127.18.96.190:46593 requested a full tablet report, sending...
I20260928 14:50:58.344740 2268 ts_manager.cc:194] Registered new tserver with Master: bb1b416518bf4bc2898e5cb618b99fbe (127.18.96.129:38457)
I20260928 14:50:58.346105 2268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:47913
I20260928 14:50:58.347261 2457 heartbeater.cc:503] Master 127.18.96.190:46593 was elected leader, sending a full tablet report...
Sep 28 14:50:58 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607058, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.129@KRBTEST.COM
I20260928 14:50:58.355257 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.345532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.18.96.129:38457)
0928 14:50:58.345757 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.345763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.345774 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:50:58.345892 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.345894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.345913 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.346002 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:58.346544 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.347606 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.348629 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.348949 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.349042 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.351383 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:58.351392 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:58.351393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:58.351402 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:58.353569 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.354154 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.354157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.354157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.354224 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.354481 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.354482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.354482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.354687 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:58.354803 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.355028 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.355149 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260928 14:50:58.355257 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.345473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38457 (local address 127.0.0.1:56592)
0928 14:50:58.345668 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:58.345706 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:58.345782 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:58.346004 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:58.346009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:58.346369 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:58.346453 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.346456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.347688 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.347690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.348530 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.348534 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.349045 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.349046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.349150 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.349699 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:58.349704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:58.351277 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:58.353696 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:58.353706 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:58.353709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:58.353965 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:58.354327 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:58.354328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:58.354329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:58.354404 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:58.354776 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:58.354786 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:58.354852 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 14:50:58.354962 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:50:58.355139 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":128,"threads_started":1}
I20260928 14:50:58.356087 2378 tablet_service.cc:1514] Processing DeleteTablet for tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:38 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:56592
I20260928 14:50:58.356750 2466 tablet_replica.cc:330] stopping tablet replica
I20260928 14:50:58.356887 2466 raft_consensus.cc:2229] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:58.356978 2466 raft_consensus.cc:2258] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:58.357323 2466 ts_tablet_manager.cc:1905] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:58.359828 2466 ts_tablet_manager.cc:1918] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:58.359964 2466 log.cc:1196] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal/wals/082c5a8dcb1b4b248a2b4bc8c9d3b8a0
I20260928 14:50:58.360303 2466 ts_tablet_manager.cc:1939] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P bb1b416518bf4bc2898e5cb618b99fbe: Deleting consensus metadata
I20260928 14:50:58.360889 2253 catalog_manager.cc:4841] TS bb1b416518bf4bc2898e5cb618b99fbe (127.18.96.129:38457): tablet 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 (table test-table [id=fc1895dbedd647b18edcac205d34c890]) successfully deleted
W20260928 14:50:58.427173 2462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:58.427464 2462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:58.427506 2462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:58.428766 2462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:58.428855 2462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:58.428907 2462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:50:58.430464 2462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:41927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=32999
--tserver_master_addrs=127.18.96.190:46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:58.430768 2462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:58.431077 2462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:58.436563 2474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:58.436581 2472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:58.436575 2471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:58.436769 2462 server_base.cc:1034] running on GCE node
I20260928 14:50:58.436985 2462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:58.437273 2462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:58.438468 2462 hybrid_clock.cc:648] HybridClock initialized: now 1790607058438449 us; error 36 us; skew 500 ppm
Sep 28 14:50:58 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607058, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:58.441288 2462 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:50:58.441807 2462 webserver.cc:460] Webserver started at http://127.18.96.130:32999/ using document root <none> and password file <none>
I20260928 14:50:58.442040 2462 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:58.442098 2462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:58.443567 2462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:58.444542 2481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:58.444800 2462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:50:58.444895 2462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "6dd630eee3584b48be15995727ebe03c"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "c0783437c5e14ea47b7a4b0dac949356"
server_key_iv: "8d8d307a72968dd5e20aa87d8479c307"
server_key_version: "encryptionkey@0"
I20260928 14:50:58.445338 2462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:58.463552 2462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:58.465939 2462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:58.466104 2462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:58.466310 2462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:58.466817 2488 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 14:50:58.468427 2488 ts_tablet_manager.cc:1852] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 14:50:58.468729 2488 ts_tablet_manager.cc:1905] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:58.471205 2488 ts_tablet_manager.cc:1918] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:58.471341 2488 log.cc:1196] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/082c5a8dcb1b4b248a2b4bc8c9d3b8a0
I20260928 14:50:58.471678 2488 ts_tablet_manager.cc:1939] T 082c5a8dcb1b4b248a2b4bc8c9d3b8a0 P 6dd630eee3584b48be15995727ebe03c: Deleting consensus metadata
I20260928 14:50:58.472033 2462 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 14:50:58.472086 2462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 14:50:58.472116 2462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:58.472165 2462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:58.478250 2462 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:41927
I20260928 14:50:58.478356 2596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:41927 every 8 connection(s)
I20260928 14:50:58.478667 2462 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:58.484407 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 2462
I20260928 14:50:58.484699 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:34879
--local_ip_for_outbound_sockets=127.18.96.131
--tserver_master_addrs=127.18.96.190:46593
--webserver_port=34447
--webserver_interface=127.18.96.131
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:50:58 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607058, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:58.489212 2460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.479532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:49509 (local address 127.18.96.190:46593)
0928 14:50:58.479604 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.479606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.479817 (+ 211us) server_negotiation.cc:407] Connection header received
0928 14:50:58.480562 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.480563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.480578 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.480639 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:58.481357 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.481878 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.483036 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.483272 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.483340 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.485856 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:58.485859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:58.485859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:58.485869 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:58.487467 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.488061 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.488063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.488063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.488121 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.488451 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.488452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.488452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.488608 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:58.488662 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.489054 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.489144 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:50:58.489543 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.479511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:49509)
0928 14:50:58.479683 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:58.479721 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:58.480417 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:58.480731 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:58.480734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:58.480989 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:58.481247 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.481251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.481987 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.481987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.482872 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.482875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.483364 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.483365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.483478 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.484265 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:58.484273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:58.485751 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:58.487568 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:58.487575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:58.487578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:58.487966 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:58.488192 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:58.488193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:58.488194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:58.488353 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:58.488681 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:58.488689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:58.488885 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 14:50:58.489009 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:50:58.489231 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260928 14:50:58.490155 2597 heartbeater.cc:347] Connected to a master server at 127.18.96.190:46593
I20260928 14:50:58.490273 2597 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:58.490490 2597 heartbeater.cc:511] Master 127.18.96.190:46593 requested a full tablet report, sending...
I20260928 14:50:58.490912 2268 ts_manager.cc:194] Registered new tserver with Master: 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927)
I20260928 14:50:58.491331 2268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:49509
W20260928 14:50:58.571759 2600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:58.572037 2600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:58.572064 2600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:58.573318 2600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:58.573366 2600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:58.573395 2600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:50:58.574919 2600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:34879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=34447
--tserver_master_addrs=127.18.96.190:46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:50:58.575219 2600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:58.575464 2600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:58.581439 2605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:58.581434 2608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:58.581624 2606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:58.581516 2600 server_base.cc:1034] running on GCE node
I20260928 14:50:58.581936 2600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:58.582226 2600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:58.583380 2600 hybrid_clock.cc:648] HybridClock initialized: now 1790607058583363 us; error 41 us; skew 500 ppm
Sep 28 14:50:58 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607058, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:58.586050 2600 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:50:58.586565 2600 webserver.cc:460] Webserver started at http://127.18.96.131:34447/ using document root <none> and password file <none>
I20260928 14:50:58.586808 2600 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:58.586894 2600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:58.588656 2600 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:58.589416 2615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:58.589658 2600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:58.589773 2600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "3786aa5155a34d32a1489c4ab695d12a"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "b0c9d7b7a3f2aafac34572315531c057"
server_key_iv: "feaf337565ace9f5a12b0b41e5bb7f8a"
server_key_version: "encryptionkey@0"
I20260928 14:50:58.590175 2600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:58.622067 2600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:58.624634 2600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:58.624764 2600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:58.624969 2600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:58.625360 2600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:58.625398 2600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:58.625422 2600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:58.625434 2600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:58.632647 2600 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:34879
I20260928 14:50:58.632721 2728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:34879 every 8 connection(s)
I20260928 14:50:58.633157 2600 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:50:58 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607058, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:50:58.641127 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 2600
I20260928 14:50:58.644383 2460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.634000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:33287 (local address 127.18.96.190:46593)
0928 14:50:58.634079 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.634082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.634331 (+ 249us) server_negotiation.cc:407] Connection header received
0928 14:50:58.634832 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.634833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.634843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.634910 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:58.636068 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.636625 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.637694 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.637893 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.637964 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.640453 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:58.640455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:58.640456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:58.640464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:58.642529 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.643211 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.643213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.643213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.643282 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.643567 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.643568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.643569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.643692 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:58.643771 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.644193 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.644281 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:50:58.644619 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.633970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:33287)
0928 14:50:58.634142 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:58.634192 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:58.634712 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:58.634976 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:58.634979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:58.635564 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:58.635814 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.635819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.636728 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.636729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.637564 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.637566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.638008 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.638009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.638158 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.638675 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:58.638684 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:58.640319 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:58.642635 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:58.642644 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:58.642646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:58.643037 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:58.643352 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:58.643353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:58.643354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:58.643488 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:58.643762 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:58.643769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:58.644021 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 14:50:58.644169 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 14:50:58.644358 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":158,"threads_started":1}
I20260928 14:50:58.645133 2729 heartbeater.cc:347] Connected to a master server at 127.18.96.190:46593
I20260928 14:50:58.645272 2729 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:58.645483 2729 heartbeater.cc:511] Master 127.18.96.190:46593 requested a full tablet report, sending...
I20260928 14:50:58.645942 2268 ts_manager.cc:194] Registered new tserver with Master: 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
I20260928 14:50:58.646384 2268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:33287
I20260928 14:50:58.649489 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.641619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.18.96.190:46593)
0928 14:50:58.641796 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.641798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.641857 (+ 59us) server_negotiation.cc:407] Connection header received
0928 14:50:58.641930 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.641930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.641950 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.642018 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:58.642849 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.643528 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.644659 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.645011 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.645075 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.646473 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:58.646474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:58.646475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:58.646483 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:58.648171 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.648591 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.648592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.648593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.648654 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.648861 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.648861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.648862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.648977 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:58.649193 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.649231 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.649319 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260928 14:50:58.650309 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:58.653093 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.650658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.18.96.190:46593)
0928 14:50:58.650695 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.650696 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.650820 (+ 124us) server_negotiation.cc:407] Connection header received
0928 14:50:58.650872 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.650873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.650902 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.650961 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:58.651074 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.651573 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.652461 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.652682 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.652735 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.652881 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:58.652924 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 14:50:58.653048 (+ 124us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58110: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 14:50:58.653136 2733 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58110: token expired
W20260928 14:50:58.653388 2193 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.96.190:46593)
I20260928 14:50:58.659318 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.653656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.18.96.190:46593)
0928 14:50:58.653690 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.653691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.653698 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:50:58.653743 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.653744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.653751 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.653792 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:58.654223 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.654666 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.655386 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.655578 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.655632 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.656858 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:58.656860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:58.656860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:58.656866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:58.658164 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.658481 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.658482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.658483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.658534 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:58.658773 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:58.658774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:58.658774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:58.658874 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:58.658945 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.659144 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.659207 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 14:50:58.660633 2268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:58.662314 2268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:58.665009 2378 tablet_service.cc:1467] Processing CreateTablet for tablet d46dc87b15144a3b872f6653649a0d83 (DEFAULT_TABLE table=test-table [id=50b55dec9420407aba414f515112aa6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:58.665325 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46dc87b15144a3b872f6653649a0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:58.666764 2348 tablet_bootstrap.cc:492] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Bootstrap starting.
I20260928 14:50:58.667098 2348 tablet_bootstrap.cc:654] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:58.667649 2348 tablet_bootstrap.cc:492] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: No bootstrap required, opened a new log
I20260928 14:50:58.667697 2348 ts_tablet_manager.cc:1397] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:50:58.668081 2348 raft_consensus.cc:348] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:50:58.668154 2348 raft_consensus.cc:374] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:58.668197 2348 raft_consensus.cc:729] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1b416518bf4bc2898e5cb618b99fbe, State: Initialized, Role: FOLLOWER
I20260928 14:50:58.668249 2348 consensus_queue.cc:260] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:50:58.668334 2348 ts_tablet_manager.cc:1428] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:50:58.668715 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.664348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:34879 (local address 127.0.0.1:49052)
0928 14:50:58.664410 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:58.664423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:58.664470 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:58.665056 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:58.665057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:58.665059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:58.665103 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.665104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.666159 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.666160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.667407 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.667410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.668418 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.668419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.668499 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.668526 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:50:58.668591 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:50:58.668643 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:50:58.668773 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.664579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.18.96.131:34879)
0928 14:50:58.664894 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.664899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.664910 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:50:58.664948 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.664950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.664966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.665058 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:58.665196 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.666070 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.667496 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.668331 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.668417 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.668486 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.668613 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.668655 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":96,"threads_started":1}
I20260928 14:50:58.669376 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.664482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.18.96.130:41927)
0928 14:50:58.664710 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.664715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.664724 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:50:58.664767 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.664768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.664781 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.664881 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:58.665212 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.666263 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.667645 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.668801 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.669140 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.669198 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.669282 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.669317 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":135,"threads_started":1}
I20260928 14:50:58.669381 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.664350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:41927 (local address 127.0.0.1:51324)
0928 14:50:58.664555 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:58.664564 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:58.664610 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:58.664918 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:58.664920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:58.664923 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:58.664975 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.664977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.666449 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.666450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.667488 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.667490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.669138 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.669139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.669203 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.669207 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:50:58.669273 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:50:58.669312 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":21,"thread_start_us":76,"threads_started":1}
I20260928 14:50:58.669549 2663 tablet_service.cc:1467] Processing CreateTablet for tablet d46dc87b15144a3b872f6653649a0d83 (DEFAULT_TABLE table=test-table [id=50b55dec9420407aba414f515112aa6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:58.669971 2663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46dc87b15144a3b872f6653649a0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:58.669931 2531 tablet_service.cc:1467] Processing CreateTablet for tablet d46dc87b15144a3b872f6653649a0d83 (DEFAULT_TABLE table=test-table [id=50b55dec9420407aba414f515112aa6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:58.670220 2531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46dc87b15144a3b872f6653649a0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:58.672526 2488 tablet_bootstrap.cc:492] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: Bootstrap starting.
I20260928 14:50:58.673501 2488 tablet_bootstrap.cc:654] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:58.673659 2739 tablet_bootstrap.cc:492] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: Bootstrap starting.
I20260928 14:50:58.673874 2488 log.cc:824] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:58.674417 2739 tablet_bootstrap.cc:654] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:58.674751 2488 tablet_bootstrap.cc:492] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: No bootstrap required, opened a new log
I20260928 14:50:58.674832 2488 ts_tablet_manager.cc:1397] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:50:58.675081 2739 log.cc:824] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:58.675941 2739 tablet_bootstrap.cc:492] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: No bootstrap required, opened a new log
I20260928 14:50:58.676033 2739 ts_tablet_manager.cc:1397] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:50:58.677177 2488 raft_consensus.cc:348] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:50:58.677378 2488 raft_consensus.cc:374] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:58.677419 2488 raft_consensus.cc:729] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd630eee3584b48be15995727ebe03c, State: Initialized, Role: FOLLOWER
I20260928 14:50:58.677567 2488 consensus_queue.cc:260] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:50:58.677866 2597 heartbeater.cc:503] Master 127.18.96.190:46593 was elected leader, sending a full tablet report...
I20260928 14:50:58.677873 2488 ts_tablet_manager.cc:1428] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:58.678251 2739 raft_consensus.cc:348] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:50:58.678434 2739 raft_consensus.cc:374] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:58.678501 2739 raft_consensus.cc:729] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3786aa5155a34d32a1489c4ab695d12a, State: Initialized, Role: FOLLOWER
I20260928 14:50:58.678645 2739 consensus_queue.cc:260] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:50:58.678983 2729 heartbeater.cc:503] Master 127.18.96.190:46593 was elected leader, sending a full tablet report...
I20260928 14:50:58.678992 2739 ts_tablet_manager.cc:1428] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 14:50:58.729346 2598 tablet.cc:2369] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:58.815212 2463 raft_consensus.cc:482] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:58.815379 2463 raft_consensus.cc:504] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:50:58.815935 2463 leader_election.cc:288] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927), 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
I20260928 14:50:58.820488 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.816326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:34183 (local address 127.18.96.131:34879)
0928 14:50:58.816372 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.816374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.816422 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:50:58.816533 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.816534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.816545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.816613 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:58.816763 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.817655 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.819297 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.820082 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.820155 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.820164 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.820363 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.820408 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:50:58.820529 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.816241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:37329 (local address 127.18.96.130:41927)
0928 14:50:58.816295 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.816298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.816310 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:50:58.816398 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.816399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.816413 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.816519 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:58.816774 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.817602 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.819299 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.819994 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.820101 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.820113 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.820363 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.820408 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:50:58.820508 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.816159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:41927 (local address 127.18.96.129:37329)
0928 14:50:58.816234 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:58.816248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:58.816312 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:58.816545 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:58.816545 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:58.816547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:58.816659 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.816662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.817740 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.817741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.819176 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.819181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.820108 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.820109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.820232 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.820268 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:50:58.820333 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:50:58.820401 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":130}
I20260928 14:50:58.820523 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.816223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:34879 (local address 127.18.96.129:34183)
0928 14:50:58.816356 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:58.816365 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:58.816436 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:58.816625 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:58.816625 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:58.816627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:58.816687 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.816689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.817751 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.817751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.819176 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:58.819181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.820207 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.820208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:58.820260 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.820271 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 14:50:58.820333 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 14:50:58.820401 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":79,"thread_start_us":74,"threads_started":1}
I20260928 14:50:58.820986 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46dc87b15144a3b872f6653649a0d83" candidate_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3786aa5155a34d32a1489c4ab695d12a" is_pre_election: true
I20260928 14:50:58.820986 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46dc87b15144a3b872f6653649a0d83" candidate_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd630eee3584b48be15995727ebe03c" is_pre_election: true
I20260928 14:50:58.821159 2683 raft_consensus.cc:2454] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1b416518bf4bc2898e5cb618b99fbe in term 0.
I20260928 14:50:58.821159 2551 raft_consensus.cc:2454] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1b416518bf4bc2898e5cb618b99fbe in term 0.
I20260928 14:50:58.821411 2343 leader_election.cc:302] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd630eee3584b48be15995727ebe03c, bb1b416518bf4bc2898e5cb618b99fbe; no voters:
I20260928 14:50:58.821609 2463 raft_consensus.cc:2781] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:58.821683 2463 raft_consensus.cc:482] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:58.821703 2463 raft_consensus.cc:3037] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:58.822546 2463 raft_consensus.cc:504] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:50:58.822713 2463 leader_election.cc:288] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 1 election: Requested vote from peers 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927), 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
I20260928 14:50:58.822916 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46dc87b15144a3b872f6653649a0d83" candidate_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3786aa5155a34d32a1489c4ab695d12a"
I20260928 14:50:58.822937 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46dc87b15144a3b872f6653649a0d83" candidate_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd630eee3584b48be15995727ebe03c"
I20260928 14:50:58.823010 2683 raft_consensus.cc:3037] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:58.823058 2551 raft_consensus.cc:3037] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:58.823817 2683 raft_consensus.cc:2454] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1b416518bf4bc2898e5cb618b99fbe in term 1.
I20260928 14:50:58.824014 2551 raft_consensus.cc:2454] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1b416518bf4bc2898e5cb618b99fbe in term 1.
I20260928 14:50:58.824067 2342 leader_election.cc:302] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3786aa5155a34d32a1489c4ab695d12a, bb1b416518bf4bc2898e5cb618b99fbe; no voters:
I20260928 14:50:58.824343 2463 raft_consensus.cc:2781] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:58.824559 2463 raft_consensus.cc:686] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 LEADER]: Becoming Leader. State: Replica: bb1b416518bf4bc2898e5cb618b99fbe, State: Running, Role: LEADER
I20260928 14:50:58.824658 2463 consensus_queue.cc:237] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:50:58.825691 2268 catalog_manager.cc:5495] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to bb1b416518bf4bc2898e5cb618b99fbe (127.18.96.129). New cstate: current_term: 1 leader_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } health_report { overall_health: HEALTHY } } }
W20260928 14:50:58.832479 2458 tablet.cc:2369] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:58.852248 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.849648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.18.96.129:38457)
0928 14:50:58.849692 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:58.849694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:58.849703 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:50:58.849793 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:58.849794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:58.849813 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:58.849889 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:58.850025 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.850524 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.851330 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:58.851577 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:58.851648 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:58.851823 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:58.851952 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:58.852014 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:58.852124 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 14:50:58.852156 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:50:58.853997 2551 raft_consensus.cc:1261] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [term 1 FOLLOWER]: Refusing update from remote peer bb1b416518bf4bc2898e5cb618b99fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:58.854074 2683 raft_consensus.cc:1261] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [term 1 FOLLOWER]: Refusing update from remote peer bb1b416518bf4bc2898e5cb618b99fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:58.854440 2745 consensus_queue.cc:1035] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:58.854640 2463 consensus_queue.cc:1035] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:58.859918 2267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:58.860147 2267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:58.861243 2267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Sending DeleteTablet for 3 replicas of tablet d46dc87b15144a3b872f6653649a0d83
I20260928 14:50:58.861652 2531 tablet_service.cc:1514] Processing DeleteTablet for tablet d46dc87b15144a3b872f6653649a0d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:58 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:51324
I20260928 14:50:58.861671 2663 tablet_service.cc:1514] Processing DeleteTablet for tablet d46dc87b15144a3b872f6653649a0d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:58 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:49052
I20260928 14:50:58.861680 2378 tablet_service.cc:1514] Processing DeleteTablet for tablet d46dc87b15144a3b872f6653649a0d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:58 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:56592
I20260928 14:50:58.861969 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 2306
I20260928 14:50:58.862634 2759 tablet_replica.cc:330] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: stopping tablet replica
I20260928 14:50:58.862761 2759 raft_consensus.cc:2229] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:58.862900 2759 raft_consensus.cc:2258] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:58.863586 2759 ts_tablet_manager.cc:1905] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:58.863945 2758 tablet_replica.cc:330] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: stopping tablet replica
I20260928 14:50:58.864089 2758 raft_consensus.cc:2229] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:58.864220 2758 raft_consensus.cc:2258] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:58.865794 2759 ts_tablet_manager.cc:1918] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:58.865855 2759 log.cc:1196] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/d46dc87b15144a3b872f6653649a0d83
I20260928 14:50:58.866112 2759 ts_tablet_manager.cc:1939] T d46dc87b15144a3b872f6653649a0d83 P 6dd630eee3584b48be15995727ebe03c: Deleting consensus metadata
I20260928 14:50:58.866591 2253 catalog_manager.cc:4841] TS 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927): tablet d46dc87b15144a3b872f6653649a0d83 (table test-table [id=50b55dec9420407aba414f515112aa6b]) successfully deleted
W20260928 14:50:58.867857 2253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:38457 (error 108)
I20260928 14:50:58.868490 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:58.868344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38457 (local address 127.0.0.1:56604)
0928 14:50:58.868394 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:58.868449 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:38457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 14:50:58.868643 2758 ts_tablet_manager.cc:1905] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:58.868785 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 2462
W20260928 14:50:58.868643 2253 catalog_manager.cc:4584] TS bb1b416518bf4bc2898e5cb618b99fbe (127.18.96.129:38457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d46dc87b15144a3b872f6653649a0d83: Network error: Client connection negotiation failed: client connection to 127.18.96.129:38457: connect: Connection refused (error 111)
I20260928 14:50:58.870771 2758 ts_tablet_manager.cc:1918] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:58.870857 2758 log.cc:1196] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/d46dc87b15144a3b872f6653649a0d83
I20260928 14:50:58.871170 2758 ts_tablet_manager.cc:1939] T d46dc87b15144a3b872f6653649a0d83 P 3786aa5155a34d32a1489c4ab695d12a: Deleting consensus metadata
I20260928 14:50:58.871663 2252 catalog_manager.cc:4841] TS 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879): tablet d46dc87b15144a3b872f6653649a0d83 (table test-table [id=50b55dec9420407aba414f515112aa6b]) successfully deleted
I20260928 14:50:58.874269 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 2600
I20260928 14:50:58.879143 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 2235
I20260928 14:51:18.885442 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:46593
--webserver_interface=127.18.96.190
--webserver_port=43553
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:18.977612 2761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:18.977841 2761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:18.977866 2761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:18.978984 2761 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:51:18.979022 2761 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:51:18.979035 2761 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:51:18.979046 2761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:18.979059 2761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:18.979074 2761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:18.979084 2761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:18.979096 2761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:18.980624 2761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:46593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=43553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:18.980831 2761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:18.981003 2761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:18.986109 2767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:18.986136 2761 server_base.cc:1034] running on GCE node
W20260928 14:51:18.986151 2768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:18.986100 2770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:18.986686 2761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:18.986949 2761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:18.988121 2761 hybrid_clock.cc:648] HybridClock initialized: now 1790607078988123 us; error 50 us; skew 500 ppm
Sep 28 14:51:18 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607078, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:18.997797 2761 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:51:18.998315 2761 webserver.cc:460] Webserver started at http://127.18.96.190:43553/ using document root <none> and password file <none>
I20260928 14:51:18.998538 2761 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:18.998587 2761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:19.000312 2761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:51:19.001118 2777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:19.001353 2761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:19.001464 2761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "9600b5ccece148c9b7d72203719e8ad6"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "4941c8fb0876a176bc88046bc0386b73"
server_key_iv: "21cd08e92b55a72fbeba391e1297ce8e"
server_key_version: "encryptionkey@0"
I20260928 14:51:19.001868 2761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:19.035099 2761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:19.037513 2761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:19.037645 2761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:19.042275 2761 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:46593
I20260928 14:51:19.042312 2829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:46593 every 8 connection(s)
I20260928 14:51:19.042759 2761 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:19.044489 2830 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:51:19.045176 2830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Bootstrap starting.
I20260928 14:51:19.051724 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 2761
I20260928 14:51:19.052119 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:38457
--local_ip_for_outbound_sockets=127.18.96.129
--tserver_master_addrs=127.18.96.190:46593
--webserver_port=33605
--webserver_interface=127.18.96.129
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:51:19.054735 2830 log.cc:824] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:19.058383 2830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:51:19.058801 2830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Bootstrap complete.
I20260928 14:51:19.063077 2830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:51:19.063530 2830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9600b5ccece148c9b7d72203719e8ad6, State: Initialized, Role: FOLLOWER
I20260928 14:51:19.063736 2830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:51:19.063830 2830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:19.063916 2830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:19.063984 2830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:51:19.065568 2830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:51:19.065693 2830 leader_election.cc:302] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9600b5ccece148c9b7d72203719e8ad6; no voters:
I20260928 14:51:19.065862 2830 leader_election.cc:288] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:51:19.065980 2834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:51:19.066134 2830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:19.066208 2834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [term 3 LEADER]: Becoming Leader. State: Replica: 9600b5ccece148c9b7d72203719e8ad6, State: Running, Role: LEADER
I20260928 14:51:19.066355 2834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } }
I20260928 14:51:19.066772 2835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9600b5ccece148c9b7d72203719e8ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } } }
I20260928 14:51:19.066874 2835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:19.067034 2836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9600b5ccece148c9b7d72203719e8ad6. Latest consensus state: current_term: 3 leader_uuid: "9600b5ccece148c9b7d72203719e8ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9600b5ccece148c9b7d72203719e8ad6" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 46593 } } }
I20260928 14:51:19.067077 2844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:19.067101 2836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:19.068130 2844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:19.068284 2844 catalog_manager.cc:1259] Loaded cluster ID: f07a4da4b44e44a98486401de377404a
I20260928 14:51:19.068514 2844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:19.069821 2844 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:19.069993 2844 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Loaded TSK: 0
I20260928 14:51:19.070034 2844 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9600b5ccece148c9b7d72203719e8ad6: Loaded TSK: 1
F20260928 14:51:19.070274 2844 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb3a08bf9ad google::LogMessage::Fail() at ??:0
@ 0x7fb3a08c288a google::LogMessage::SendToLog() at ??:0
@ 0x7fb3a08bf4de google::LogMessage::Flush() at ??:0
@ 0x7fb3a08c0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb3a302c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb3a3029d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb3a88191cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb3a881948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb3a8819594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb3a87dfc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb3a87e9ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb3a278d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb3a2782954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb3a35e66db start_thread at ??:0
@ 0x7fb39f3bd61f clone at ??:0
W20260928 14:51:19.173602 2832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:19.173804 2832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:19.173830 2832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:19.175016 2832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:19.175066 2832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:19.175100 2832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:51:19.176720 2832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:38457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=33605
--tserver_master_addrs=127.18.96.190:46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:19.176990 2832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:19.177254 2832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:19.182606 2873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:19.182590 2875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:19.182826 2832 server_base.cc:1034] running on GCE node
W20260928 14:51:19.182618 2872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:19.183089 2832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:19.183358 2832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:19.184526 2832 hybrid_clock.cc:648] HybridClock initialized: now 1790607079184502 us; error 41 us; skew 500 ppm
Sep 28 14:51:19 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607079, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:19.187232 2832 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:51:19.187736 2832 webserver.cc:460] Webserver started at http://127.18.96.129:33605/ using document root <none> and password file <none>
I20260928 14:51:19.188040 2832 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:19.188113 2832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:19.189679 2832 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:19.190508 2882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:19.190765 2832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:19.190893 2832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "bb1b416518bf4bc2898e5cb618b99fbe"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "661636a53aa3eb86dc43e0c7ce17d14d"
server_key_iv: "7c82e3d068a3fc26d342478814202f92"
server_key_version: "encryptionkey@0"
I20260928 14:51:19.191344 2832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:19.214754 2832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:19.217669 2832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:19.217870 2832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:19.218155 2832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:19.218762 2889 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:51:19.221060 2832 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:51:19.221165 2832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 14:51:19.221220 2832 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:51:19.221905 2832 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:51:19.221995 2832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 14:51:19.222069 2889 tablet_bootstrap.cc:492] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Bootstrap starting.
I20260928 14:51:19.229620 2832 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:38457
I20260928 14:51:19.229795 2997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:38457 every 8 connection(s)
I20260928 14:51:19.230077 2832 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:51:19.235713 2889 log.cc:824] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:19.238900 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 2832
I20260928 14:51:19.239006 2889 tablet_bootstrap.cc:492] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:51:19.239280 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:41927
--local_ip_for_outbound_sockets=127.18.96.130
--tserver_master_addrs=127.18.96.190:46593
--webserver_port=32999
--webserver_interface=127.18.96.130
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:51:19.239395 2889 tablet_bootstrap.cc:492] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Bootstrap complete.
I20260928 14:51:19.240871 2889 ts_tablet_manager.cc:1397] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.010s
I20260928 14:51:19.242573 2889 raft_consensus.cc:348] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:51:19.242946 2889 raft_consensus.cc:729] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1b416518bf4bc2898e5cb618b99fbe, State: Initialized, Role: FOLLOWER
I20260928 14:51:19.243152 2889 consensus_queue.cc:260] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:51:19.243494 2889 ts_tablet_manager.cc:1428] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
W20260928 14:51:19.307191 3000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:51:19.230850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:48015)
0928 14:51:19.231117 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:19.231166 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:19.231659 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:19.306954 (+ 75295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
W20260928 14:51:19.307456 2998 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:46593 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.190:46593: Client connection negotiation failed: client connection to 127.18.96.190:46593: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:51:19.307843 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:19.307721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:51143)
0928 14:51:19.307780 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:19.307823 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 14:51:19.308244 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:19.308145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:57611)
0928 14:51:19.308199 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:19.308226 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 14:51:19.308390 2998 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:51:19.328330 3002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:19.328545 3002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:19.328572 3002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:19.329803 3002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:19.329849 3002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:19.329876 3002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:51:19.331389 3002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:41927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=32999
--tserver_master_addrs=127.18.96.190:46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:19.331655 3002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:19.331936 3002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:19.337198 3011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:19.337198 3008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:19.337363 3009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:19.337419 3002 server_base.cc:1034] running on GCE node
I20260928 14:51:19.337687 3002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:19.337935 3002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:19.339107 3002 hybrid_clock.cc:648] HybridClock initialized: now 1790607079339081 us; error 37 us; skew 500 ppm
Sep 28 14:51:19 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607079, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:19.341817 3002 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:51:19.342351 3002 webserver.cc:460] Webserver started at http://127.18.96.130:32999/ using document root <none> and password file <none>
I20260928 14:51:19.342597 3002 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:19.342679 3002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:19.344246 3002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:19.344944 3018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:19.345253 3002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:19.345368 3002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "6dd630eee3584b48be15995727ebe03c"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "c0783437c5e14ea47b7a4b0dac949356"
server_key_iv: "8d8d307a72968dd5e20aa87d8479c307"
server_key_version: "encryptionkey@0"
I20260928 14:51:19.345819 3002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:19.375034 3002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:19.377625 3002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:19.377787 3002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:19.378057 3002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:19.378417 3002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:19.378475 3002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:19.378512 3002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:19.378564 3002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:19.384827 3002 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:41927
I20260928 14:51:19.384883 3132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:41927 every 8 connection(s)
I20260928 14:51:19.385284 3002 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:51:19.386451 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:19.386016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:55847)
0928 14:51:19.386220 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:19.386298 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
W20260928 14:51:19.386960 3133 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:46593 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.190:46593: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
I20260928 14:51:19.386969 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 3002
I20260928 14:51:19.387241 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:34879
--local_ip_for_outbound_sockets=127.18.96.131
--tserver_master_addrs=127.18.96.190:46593
--webserver_port=34447
--webserver_interface=127.18.96.131
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:51:19.387364 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:19.387232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:46543)
0928 14:51:19.387286 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:19.387314 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:51:19.387789 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:19.387692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:40049)
0928 14:51:19.387743 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:19.387766 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:51:19.387934 3133 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:51:19.471643 3136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:19.471901 3136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:19.471971 3136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:19.473310 3136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:19.473363 3136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:19.473390 3136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:51:19.474910 3136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:34879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=34447
--tserver_master_addrs=127.18.96.190:46593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:19.475183 3136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:19.475433 3136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:19.480682 3141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:19.480681 3142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:19.480677 3144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:19.480966 3136 server_base.cc:1034] running on GCE node
I20260928 14:51:19.481235 3136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:19.481495 3136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:19.482679 3136 hybrid_clock.cc:648] HybridClock initialized: now 1790607079482649 us; error 39 us; skew 500 ppm
Sep 28 14:51:19 dist-test-slave-j968 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607079, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:19.485375 3136 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:51:19.485863 3136 webserver.cc:460] Webserver started at http://127.18.96.131:34447/ using document root <none> and password file <none>
I20260928 14:51:19.486097 3136 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:19.486150 3136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:19.487548 3136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:19.488229 3151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:19.488447 3136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:19.488523 3136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "3786aa5155a34d32a1489c4ab695d12a"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-j968"
server_key: "b0c9d7b7a3f2aafac34572315531c057"
server_key_iv: "feaf337565ace9f5a12b0b41e5bb7f8a"
server_key_version: "encryptionkey@0"
I20260928 14:51:19.488888 3136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:19.503293 3136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:19.506078 3136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:19.506215 3136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:19.506547 3136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:19.506902 3136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:19.506942 3136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:19.506968 3136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:19.506979 3136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:19.513326 3136 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:34879
I20260928 14:51:19.513473 3265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:34879 every 8 connection(s)
I20260928 14:51:19.513794 3136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:51:19.514935 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:19.514514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:48513)
0928 14:51:19.514713 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:19.514790 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
W20260928 14:51:19.515424 3266 heartbeater.cc:644] Failed to heartbeat to 127.18.96.190:46593 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.190:46593: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
I20260928 14:51:19.515802 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:19.515667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:34433)
0928 14:51:19.515748 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:19.515781 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 14:51:19.516316 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:19.516193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:36627)
0928 14:51:19.516262 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:19.516290 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 14:51:19.516541 3266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:51:19.522894 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 3136
I20260928 14:51:20.309178 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:20.308838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:58705)
0928 14:51:20.309061 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:20.309113 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260928 14:51:20.388522 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:20.388277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:50163)
0928 14:51:20.388406 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:20.388440 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260928 14:51:20.517127 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:20.516923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:60837)
0928 14:51:20.517039 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:20.517073 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260928 14:51:20.773391 3273 raft_consensus.cc:482] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:20.773622 3273 raft_consensus.cc:504] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:51:20.774190 3273 leader_election.cc:288] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927), 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
Sep 28 14:51:20 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607079, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.130@KRBTEST.COM
Sep 28 14:51:20 dist-test-slave-j968 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607079, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.131@KRBTEST.COM
I20260928 14:51:20.785398 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:20.774313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:60227 (local address 127.18.96.131:34879)
0928 14:51:20.774519 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:20.774531 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:20.774568 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:51:20.775023 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:20.775025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:20.775064 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:20.775229 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:20.776099 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:20.777003 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:20.778219 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:20.778657 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:20.778794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:20.781758 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:20.781762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:20.781763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:20.781775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:20.783624 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:20.784185 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:20.784188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:20.784188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:20.784299 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:20.784538 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:20.784539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:20.784539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:20.784781 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:20.784877 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:20.785163 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 14:51:20.785294 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
I20260928 14:51:20.785380 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:20.774173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:34879 (local address 127.18.96.129:60227)
0928 14:51:20.774444 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:20.774496 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:20.774547 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:20.775269 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:20.775272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:20.775687 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:20.775960 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:20.775966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:20.777222 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:20.777223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:20.778068 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:20.778071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:20.778759 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:20.778760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:20.778836 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:20.779458 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:20.779467 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:20.781645 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:20.783844 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:20.783845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:20.783845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:20.784093 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:20.784382 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:20.784382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:20.784383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:20.784458 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:20.784863 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:20.784865 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:20.784938 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 14:51:20.785082 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 14:51:20.785271 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":237,"thread_start_us":50,"threads_started":1}
I20260928 14:51:20.785394 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:20.774314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:50975 (local address 127.18.96.130:41927)
0928 14:51:20.774500 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:20.774513 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:20.774554 (+ 41us) server_negotiation.cc:407] Connection header received
0928 14:51:20.775002 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:20.775004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:20.775048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:20.775240 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:20.776099 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:20.776963 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:20.778167 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:20.778593 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:20.778733 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:20.781427 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:20.781432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:20.781432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:20.781446 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:20.783560 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:20.784044 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:20.784047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:20.784048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:20.784156 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:20.784439 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:20.784440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:20.784440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:20.784710 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:20.784803 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:20.785100 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 14:51:20.785242 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260928 14:51:20.785611 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:20.774173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:41927 (local address 127.18.96.129:50975)
0928 14:51:20.774235 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:20.774247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:20.774321 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:20.775269 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:20.775272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:20.775686 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:20.775958 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:20.775966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:20.777241 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:20.777242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:20.778067 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:20.778070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:20.778706 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:20.778707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:20.778837 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:20.779516 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:20.779521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:20.781302 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:20.783654 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:20.783661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:20.783663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:20.783952 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:20.784250 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:20.784251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:20.784251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:20.784361 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:20.784793 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:20.784800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:20.784926 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 14:51:20.785027 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 14:51:20.785357 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":117}
W20260928 14:51:20.786229 2883 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
W20260928 14:51:20.786720 2884 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
I20260928 14:51:20.786772 2884 leader_election.cc:302] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1b416518bf4bc2898e5cb618b99fbe; no voters: 3786aa5155a34d32a1489c4ab695d12a, 6dd630eee3584b48be15995727ebe03c
I20260928 14:51:20.786878 3273 raft_consensus.cc:2726] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:51:21.310045 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:21.309754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:38459)
0928 14:51:21.309944 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:21.309993 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260928 14:51:21.389190 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:21.388911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:42877)
0928 14:51:21.389108 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:21.389146 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":122,"threads_started":1}
I20260928 14:51:21.517871 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:21.517663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:44067)
0928 14:51:21.517789 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:21.517828 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260928 14:51:22.311031 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:22.310587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:60697)
0928 14:51:22.310739 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:22.310786 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260928 14:51:22.374882 3281 raft_consensus.cc:482] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:22.374984 3281 raft_consensus.cc:504] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:51:22.375126 3281 leader_election.cc:288] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927), 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
W20260928 14:51:22.375612 2884 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
W20260928 14:51:22.375707 2883 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
I20260928 14:51:22.375790 2883 leader_election.cc:302] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1b416518bf4bc2898e5cb618b99fbe; no voters: 3786aa5155a34d32a1489c4ab695d12a, 6dd630eee3584b48be15995727ebe03c
I20260928 14:51:22.375913 3281 raft_consensus.cc:2726] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:51:22.389905 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:22.389697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:42113)
0928 14:51:22.389822 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:22.389863 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 14:51:22.518679 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:22.518442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:33465)
0928 14:51:22.518594 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:22.518635 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260928 14:51:23.312186 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:23.311649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:52043)
0928 14:51:23.311854 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:23.312132 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260928 14:51:23.390645 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:23.390434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:59017)
0928 14:51:23.390557 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:23.390596 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 14:51:23.519662 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:23.519378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:50687)
0928 14:51:23.519550 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:23.519595 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260928 14:51:24.313110 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:24.312835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:53905)
0928 14:51:24.312996 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:24.313058 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260928 14:51:24.317306 3288 raft_consensus.cc:482] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:24.317461 3288 raft_consensus.cc:504] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:51:24.317706 3288 leader_election.cc:288] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927), 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
W20260928 14:51:24.318161 2884 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
W20260928 14:51:24.318290 2883 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
I20260928 14:51:24.318382 2883 leader_election.cc:302] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1b416518bf4bc2898e5cb618b99fbe; no voters: 3786aa5155a34d32a1489c4ab695d12a, 6dd630eee3584b48be15995727ebe03c
I20260928 14:51:24.318481 3288 raft_consensus.cc:2726] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:51:24.391624 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:24.391268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:35041)
0928 14:51:24.391475 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:24.391572 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":153,"threads_started":1}
I20260928 14:51:24.520538 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:24.520274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:36317)
0928 14:51:24.520435 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:24.520477 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260928 14:51:25.313970 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:25.313702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:51137)
0928 14:51:25.313877 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:25.313919 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260928 14:51:25.392725 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:25.392386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:58531)
0928 14:51:25.392583 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:25.392661 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260928 14:51:25.521404 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:25.521146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:47271)
0928 14:51:25.521303 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:25.521352 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260928 14:51:26.314860 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:26.314605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:57349)
0928 14:51:26.314761 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:26.314809 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260928 14:51:26.393491 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:26.393242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:58927)
0928 14:51:26.393392 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:26.393434 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260928 14:51:26.522312 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:26.522077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:38809)
0928 14:51:26.522217 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:26.522262 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260928 14:51:26.788504 3297 raft_consensus.cc:482] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:26.788616 3297 raft_consensus.cc:504] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:51:26.788818 3297 leader_election.cc:288] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927), 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
W20260928 14:51:26.789280 2884 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
W20260928 14:51:26.789403 2883 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
I20260928 14:51:26.789546 2883 leader_election.cc:302] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1b416518bf4bc2898e5cb618b99fbe; no voters: 3786aa5155a34d32a1489c4ab695d12a, 6dd630eee3584b48be15995727ebe03c
I20260928 14:51:26.789677 3297 raft_consensus.cc:2726] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:51:27.315825 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:27.315563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:42107)
0928 14:51:27.315730 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:27.315774 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260928 14:51:27.394300 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:27.393990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:49027)
0928 14:51:27.394168 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:27.394220 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":60,"threads_started":1}
I20260928 14:51:27.523216 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:27.522909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:53571)
0928 14:51:27.523129 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:27.523168 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260928 14:51:28.316711 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:28.316442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:39817)
0928 14:51:28.316611 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:28.316661 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260928 14:51:28.395113 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:28.394804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:48679)
0928 14:51:28.394991 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:28.395049 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":61,"threads_started":1}
I20260928 14:51:28.524067 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:28.523748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:37349)
0928 14:51:28.523975 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:28.524019 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":61,"threads_started":1}
I20260928 14:51:29.317642 3304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:29.317383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:55813)
0928 14:51:29.317544 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:29.317591 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260928 14:51:29.395931 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:29.395617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:50867)
0928 14:51:29.395782 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:29.395837 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":113,"threads_started":1}
I20260928 14:51:29.524896 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:29.524668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:47509)
0928 14:51:29.524803 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:29.524850 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260928 14:51:30.318516 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:30.318246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:60715)
0928 14:51:30.318415 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:30.318465 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":118,"threads_started":1}
I20260928 14:51:30.396550 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:30.396354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:33989)
0928 14:51:30.396470 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:30.396499 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260928 14:51:30.525667 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:30.525415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:52145)
0928 14:51:30.525580 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:30.525620 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260928 14:51:31.319528 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:31.319227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:33919)
0928 14:51:31.319399 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:31.319459 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260928 14:51:31.397255 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:31.397018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:57859)
0928 14:51:31.397169 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:31.397207 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260928 14:51:31.526470 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:31.526204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:45243)
0928 14:51:31.526367 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:31.526415 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260928 14:51:32.320451 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:32.320181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:46175)
0928 14:51:32.320350 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:32.320399 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260928 14:51:32.398052 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:32.397808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:41635)
0928 14:51:32.397944 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:32.397990 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260928 14:51:32.527323 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:32.527095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:33669)
0928 14:51:32.527234 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:32.527276 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260928 14:51:33.321264 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:33.321021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:33715)
0928 14:51:33.321171 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:33.321211 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260928 14:51:33.398800 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:33.398555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:60383)
0928 14:51:33.398699 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:33.398749 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260928 14:51:33.528270 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:33.527979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:59423)
0928 14:51:33.528177 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:33.528219 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":118,"threads_started":1}
I20260928 14:51:34.302915 3319 raft_consensus.cc:482] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:34.303021 3319 raft_consensus.cc:504] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd630eee3584b48be15995727ebe03c" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 41927 } } peers { permanent_uuid: "3786aa5155a34d32a1489c4ab695d12a" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 34879 } } peers { permanent_uuid: "bb1b416518bf4bc2898e5cb618b99fbe" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38457 } }
I20260928 14:51:34.303315 3319 leader_election.cc:288] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927), 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879)
W20260928 14:51:34.303828 2883 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3786aa5155a34d32a1489c4ab695d12a (127.18.96.131:34879): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
W20260928 14:51:34.303937 2884 leader_election.cc:341] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6dd630eee3584b48be15995727ebe03c (127.18.96.130:41927): Not found: Tablet not found: d46dc87b15144a3b872f6653649a0d83
I20260928 14:51:34.303994 2884 leader_election.cc:302] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1b416518bf4bc2898e5cb618b99fbe; no voters: 3786aa5155a34d32a1489c4ab695d12a, 6dd630eee3584b48be15995727ebe03c
I20260928 14:51:34.304141 3319 raft_consensus.cc:2726] T d46dc87b15144a3b872f6653649a0d83 P bb1b416518bf4bc2898e5cb618b99fbe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:51:34.322216 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:34.321882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.129:56839)
0928 14:51:34.322118 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:34.322162 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":173,"threads_started":1}
I20260928 14:51:34.399683 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:34.399470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.130:33911)
0928 14:51:34.399595 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:34.399637 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260928 14:51:34.529201 3322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:34.528898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:46593 (local address 127.18.96.131:57657)
0928 14:51:34.529099 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:34.529152 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.190:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":142,"threads_started":1}
I20260928 14:51:34.533465 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 2832
I20260928 14:51:34.538676 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 3002
I20260928 14:51:34.543387 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 3136
2026-09-28T14:51:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58753 ms)
[----------] 30 tests from SecurityITest (237191 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:34 dist-test-slave-j968 krb5kdc[3325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:34 dist-test-slave-j968 krb5kdc[3325](info): set up 2 sockets
Sep 28 14:51:34 dist-test-slave-j968 krb5kdc[3325](info): commencing operation
krb5kdc: starting...
W20260928 14:51:35.612954 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:35 dist-test-slave-j968 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:51:35.743952 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:41845
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:41845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:35.827009 3338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:35.827294 3338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:35.827333 3338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:35.828588 3338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:35.828668 3338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:35.828691 3338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:35.828711 3338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:35.828742 3338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:35.830297 3338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:41845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:41845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:35.830592 3338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:35.830826 3338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:35.836248 3343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:35.836278 3338 server_base.cc:1034] running on GCE node
W20260928 14:51:35.836347 3346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:35.836370 3344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:35.836689 3338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:35.836957 3338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:35.838111 3338 hybrid_clock.cc:648] HybridClock initialized: now 1790607095838090 us; error 35 us; skew 500 ppm
Sep 28 14:51:35 dist-test-slave-j968 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607095, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:35.840909 3338 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:51:35.841475 3338 webserver.cc:460] Webserver started at http://127.18.96.190:41327/ using document root <none> and password file <none>
I20260928 14:51:35.841760 3338 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:35.841814 3338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:35.842003 3338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:35.843245 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "98ba348c9f54425abe2c8b56fa72bb3c"
format_stamp: "Formatted at 2026-09-28 14:51:35 on dist-test-slave-j968"
server_key: "b7597142c6476843984bd85cf19ee084"
server_key_iv: "5bc652619c0f1a556ee39ae374d1f923"
server_key_version: "encryptionkey@0"
I20260928 14:51:35.843659 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "98ba348c9f54425abe2c8b56fa72bb3c"
format_stamp: "Formatted at 2026-09-28 14:51:35 on dist-test-slave-j968"
server_key: "b7597142c6476843984bd85cf19ee084"
server_key_iv: "5bc652619c0f1a556ee39ae374d1f923"
server_key_version: "encryptionkey@0"
I20260928 14:51:35.845418 3338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:35.846321 3354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:35.846737 3338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:35.846889 3338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "98ba348c9f54425abe2c8b56fa72bb3c"
format_stamp: "Formatted at 2026-09-28 14:51:35 on dist-test-slave-j968"
server_key: "b7597142c6476843984bd85cf19ee084"
server_key_iv: "5bc652619c0f1a556ee39ae374d1f923"
server_key_version: "encryptionkey@0"
I20260928 14:51:35.846982 3338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:35.858438 3338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:35.861164 3338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:35.861349 3338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:35.865636 3338 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:41845
I20260928 14:51:35.865689 3406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:41845 every 8 connection(s)
I20260928 14:51:35.866075 3338 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:35.866989 3407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:35.869271 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 3338
I20260928 14:51:35.869365 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:35.869472 3407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c: Bootstrap starting.
I20260928 14:51:35.869638 18818 external_mini_cluster.cc:1428] Setting key 9d735b68ec6d4269b261f276dbb4caae
I20260928 14:51:35.870214 3407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:35.870554 3407 log.cc:824] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:35.871274 3407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c: No bootstrap required, opened a new log
I20260928 14:51:35.873512 3407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ba348c9f54425abe2c8b56fa72bb3c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41845 } }
I20260928 14:51:35.873701 3407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:35.873737 3407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ba348c9f54425abe2c8b56fa72bb3c, State: Initialized, Role: FOLLOWER
I20260928 14:51:35.873854 3407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ba348c9f54425abe2c8b56fa72bb3c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41845 } }
I20260928 14:51:35.873919 3407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:35.873947 3407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:35.873989 3407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:35.874619 3407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ba348c9f54425abe2c8b56fa72bb3c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41845 } }
I20260928 14:51:35.874759 3407 leader_election.cc:302] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ba348c9f54425abe2c8b56fa72bb3c; no voters:
I20260928 14:51:35.874941 3407 leader_election.cc:288] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:35.874964 3412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:35.875079 3412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [term 1 LEADER]: Becoming Leader. State: Replica: 98ba348c9f54425abe2c8b56fa72bb3c, State: Running, Role: LEADER
I20260928 14:51:35.875224 3412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ba348c9f54425abe2c8b56fa72bb3c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41845 } }
I20260928 14:51:35.875555 3407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:35.875623 3412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ba348c9f54425abe2c8b56fa72bb3c. Latest consensus state: current_term: 1 leader_uuid: "98ba348c9f54425abe2c8b56fa72bb3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ba348c9f54425abe2c8b56fa72bb3c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41845 } } }
I20260928 14:51:35.875747 3412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:35.875964 3414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98ba348c9f54425abe2c8b56fa72bb3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ba348c9f54425abe2c8b56fa72bb3c" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41845 } } }
I20260928 14:51:35.876003 3416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:35.876097 3414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:35.876734 3416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 14:51:35 dist-test-slave-j968 krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:35.878477 3416 catalog_manager.cc:1347] Generated new cluster ID: 429abab9d4c349eb9d170b67b5b4dd52
I20260928 14:51:35.878538 3416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:35.882009 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:35.870288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.18.96.190:41845)
0928 14:51:35.870464 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:35.870473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:35.870498 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:51:35.870997 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:35.871004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:35.871127 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:35.871254 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:35.872121 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:35.873111 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:35.874216 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:35.874691 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:35.874849 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:35.878214 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:35.878218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:35.878219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:35.878232 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:35.880106 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:35.880639 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:35.880643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:35.880643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:35.880750 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:35.880998 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:35.880999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:35.880999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:35.881275 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:35.881456 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:35.881587 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 14:51:35.881750 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260928 14:51:35.906136 3416 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:35.906913 3416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:35.916419 3416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c: Generated new TSK 0
I20260928 14:51:35.916656 3416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:51:35.952898 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41845
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:36.045786 3433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:36.046016 3433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:36.046052 3433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:36.047284 3433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:36.047329 3433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:36.047358 3433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:51:36.049108 3433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:36.049329 3433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:36.049525 3433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:36.055266 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.055272 3441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.055272 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:36.055588 3433 server_base.cc:1034] running on GCE node
I20260928 14:51:36.055828 3433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:36.056154 3433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:36.057325 3433 hybrid_clock.cc:648] HybridClock initialized: now 1790607096057303 us; error 38 us; skew 500 ppm
Sep 28 14:51:36 dist-test-slave-j968 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607096, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:36.060146 3433 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:51:36.060631 3433 webserver.cc:460] Webserver started at http://127.18.96.129:35233/ using document root <none> and password file <none>
I20260928 14:51:36.060892 3433 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:36.060959 3433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:36.061193 3433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:36.062306 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "c9d6a22901fb49e68649950618cc5e9a"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "d047d603686113ecfa6dd4429c8bac1e"
server_key_iv: "0a6ead2907d42b961b5f4d6a46ad5fb1"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.062662 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "c9d6a22901fb49e68649950618cc5e9a"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "d047d603686113ecfa6dd4429c8bac1e"
server_key_iv: "0a6ead2907d42b961b5f4d6a46ad5fb1"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.064129 3433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:51:36.064908 3448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.065132 3433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:36.065227 3433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "c9d6a22901fb49e68649950618cc5e9a"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "d047d603686113ecfa6dd4429c8bac1e"
server_key_iv: "0a6ead2907d42b961b5f4d6a46ad5fb1"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.065315 3433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:36.080152 3433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:36.082260 3433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:36.082427 3433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:36.082698 3433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:36.083019 3433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:36.083071 3433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.083115 3433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:36.083140 3433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.089269 3433 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:44967
I20260928 14:51:36.089334 3561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:44967 every 8 connection(s)
I20260928 14:51:36.089682 3433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:51:36 dist-test-slave-j968 krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607096, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:36.098495 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 3433
I20260928 14:51:36.098609 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:51:36.098847 18818 external_mini_cluster.cc:1428] Setting key fa6dfc29424b39c6d047fe68b6a18634
I20260928 14:51:36.102051 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.090484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:46811 (local address 127.18.96.190:41845)
0928 14:51:36.090561 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.090563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.090718 (+ 155us) server_negotiation.cc:407] Connection header received
0928 14:51:36.091319 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.091320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.091333 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.091389 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:36.092739 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.093480 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.094773 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.095124 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.095209 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.098377 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:36.098379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:36.098379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:36.098389 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:36.100214 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:36.100744 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:36.100745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:36.100745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:36.100799 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:36.101121 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:36.101158 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:36.101159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:36.101359 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:36.101454 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.101854 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.101975 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:51:36.102583 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.090436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:41845 (local address 127.18.96.129:46811)
0928 14:51:36.090588 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.090626 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:36.091210 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:36.091466 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:36.091470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:36.092293 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:36.092613 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.092632 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.093605 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.093606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.094684 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.094687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.095242 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.095243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.095431 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.096300 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:36.096310 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:36.098269 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:36.100303 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:36.100311 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:36.100315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:36.100665 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:36.100865 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:36.100866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:36.100866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:36.101021 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:36.101458 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:36.101465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:36.101675 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 14:51:36.102172 (+ 497us) client_negotiation.cc:240] Negotiation successful
0928 14:51:36.102376 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260928 14:51:36.103200 3562 heartbeater.cc:347] Connected to a master server at 127.18.96.190:41845
I20260928 14:51:36.103322 3562 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:36.103516 3562 heartbeater.cc:511] Master 127.18.96.190:41845 requested a full tablet report, sending...
I20260928 14:51:36.104241 3371 ts_manager.cc:194] Registered new tserver with Master: c9d6a22901fb49e68649950618cc5e9a (127.18.96.129:44967)
I20260928 14:51:36.105286 3371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:46811
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:51:36.127774 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41845
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:36.213795 3568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:36.214051 3568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:36.214090 3568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:36.215277 3568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:36.215346 3568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:36.215389 3568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:51:36.216995 3568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:36.217278 3568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:36.217517 3568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:36.222677 3573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.222723 3574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.222677 3576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:36.223017 3568 server_base.cc:1034] running on GCE node
I20260928 14:51:36.223197 3568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:36.223407 3568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:36.224582 3568 hybrid_clock.cc:648] HybridClock initialized: now 1790607096224567 us; error 43 us; skew 500 ppm
Sep 28 14:51:36 dist-test-slave-j968 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607096, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:36.227291 3568 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:51:36.227749 3568 webserver.cc:460] Webserver started at http://127.18.96.130:34275/ using document root <none> and password file <none>
I20260928 14:51:36.228048 3568 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:36.228118 3568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:36.228286 3568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:36.229411 3568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "0f177c3bb3e341049518c6de39e869f6"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "883dd2a7690470d68656ec4a4d598b95"
server_key_iv: "c24c8ad88052802adb5be79f9c91dd5c"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.229777 3568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "0f177c3bb3e341049518c6de39e869f6"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "883dd2a7690470d68656ec4a4d598b95"
server_key_iv: "c24c8ad88052802adb5be79f9c91dd5c"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.231160 3568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:51:36.231920 3583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.232113 3568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:36.232213 3568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "0f177c3bb3e341049518c6de39e869f6"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "883dd2a7690470d68656ec4a4d598b95"
server_key_iv: "c24c8ad88052802adb5be79f9c91dd5c"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.232300 3568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:36.245030 3568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:36.247013 3568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:36.247130 3568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:36.247319 3568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:36.247648 3568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:36.247684 3568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.247707 3568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:36.247753 3568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.253934 3568 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:43677
I20260928 14:51:36.254010 3697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:43677 every 8 connection(s)
I20260928 14:51:36.254297 3568 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:51:36 dist-test-slave-j968 krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607096, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:36.263463 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 3568
I20260928 14:51:36.263608 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:51:36.263813 18818 external_mini_cluster.cc:1428] Setting key a217f88d432e5afcac7cc6606773a1bf
I20260928 14:51:36.264204 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.255044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:50691 (local address 127.18.96.190:41845)
0928 14:51:36.255103 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.255105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.255326 (+ 221us) server_negotiation.cc:407] Connection header received
0928 14:51:36.255824 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.255825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.255835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.255968 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:36.256965 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.257520 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.258593 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.258788 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.258846 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.261070 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:36.261071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:36.261071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:36.261078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:36.262445 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:36.262911 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:36.262912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:36.262913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:36.262972 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:36.263301 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:36.263302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:36.263303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:36.263469 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:36.263606 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.264004 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.264093 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:51:36.264463 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.254985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:41845 (local address 127.18.96.130:50691)
0928 14:51:36.255196 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.255238 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:36.255735 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:36.256051 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:36.256054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:36.256672 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:36.256869 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.256875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.257674 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.257675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.258504 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.258506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.258891 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.258892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.258999 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.259545 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:36.259553 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:36.260971 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:36.262571 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:36.262578 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:36.262580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:36.262832 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:36.263046 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:36.263047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:36.263047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:36.263160 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:36.263621 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:36.263628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:36.263813 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 14:51:36.263961 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 14:51:36.264191 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260928 14:51:36.265026 3698 heartbeater.cc:347] Connected to a master server at 127.18.96.190:41845
I20260928 14:51:36.265223 3698 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:36.265410 3698 heartbeater.cc:511] Master 127.18.96.190:41845 requested a full tablet report, sending...
I20260928 14:51:36.265853 3371 ts_manager.cc:194] Registered new tserver with Master: 0f177c3bb3e341049518c6de39e869f6 (127.18.96.130:43677)
I20260928 14:51:36.266326 3371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:50691
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:51:36.290943 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41845
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:36.379456 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:36.379701 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:36.379742 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:36.381429 3703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:36.381491 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:36.381544 3703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:51:36.383116 3703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:41845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:36.383399 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:36.383639 3703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:36.388832 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.388840 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.388875 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:36.388921 3703 server_base.cc:1034] running on GCE node
I20260928 14:51:36.389328 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:36.389585 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:36.390729 3703 hybrid_clock.cc:648] HybridClock initialized: now 1790607096390720 us; error 35 us; skew 500 ppm
Sep 28 14:51:36 dist-test-slave-j968 krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607096, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:36.393522 3703 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:51:36.394013 3703 webserver.cc:460] Webserver started at http://127.18.96.131:35475/ using document root <none> and password file <none>
I20260928 14:51:36.394268 3703 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:36.394336 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:36.394484 3703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:36.395627 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "d00beb9b15cc439eacf86e55c0a180db"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "0b5e7a3c503194abe58a320434bc9b7b"
server_key_iv: "4c4fd8fdd2a5bb7cb53e26a1931d9eab"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.396030 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "d00beb9b15cc439eacf86e55c0a180db"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "0b5e7a3c503194abe58a320434bc9b7b"
server_key_iv: "4c4fd8fdd2a5bb7cb53e26a1931d9eab"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.397441 3703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:36.398236 3718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.398414 3703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:36.398515 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "d00beb9b15cc439eacf86e55c0a180db"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "0b5e7a3c503194abe58a320434bc9b7b"
server_key_iv: "4c4fd8fdd2a5bb7cb53e26a1931d9eab"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.398618 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:36.408753 3703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:36.411181 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:36.411350 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:36.411602 3703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:36.411996 3703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:36.412055 3703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.412097 3703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:36.412124 3703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.418394 3703 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:37649
I20260928 14:51:36.418438 3831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:37649 every 8 connection(s)
I20260928 14:51:36.418777 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:51:36 dist-test-slave-j968 krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607096, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:36.426795 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 3703
I20260928 14:51:36.426889 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:51:36.427129 18818 external_mini_cluster.cc:1428] Setting key 217450167a1bbe81cfa0182e1e96b151
I20260928 14:51:36.428841 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.419523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:49669 (local address 127.18.96.190:41845)
0928 14:51:36.419601 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.419604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.419783 (+ 179us) server_negotiation.cc:407] Connection header received
0928 14:51:36.420381 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.420383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.420397 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.420450 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:36.421443 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.421996 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.423019 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.423218 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.423300 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.425727 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:36.425729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:36.425730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:36.425738 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:36.427371 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:36.427813 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:36.427814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:36.427815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:36.427899 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:36.428173 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:36.428174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:36.428174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:36.428335 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:36.428404 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.428690 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.428768 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:51:36.428972 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.419483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:41845 (local address 127.18.96.131:49669)
0928 14:51:36.419638 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.419692 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:36.420272 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:36.420534 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:36.420539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:36.421134 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:36.421340 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.421345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.422096 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.422097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.422912 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.422914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.423312 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.423313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.423448 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.424104 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:36.424113 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:36.425629 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:36.427435 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:36.427444 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:36.427446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:36.427720 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:36.427952 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:36.427953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:36.427953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:36.428096 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:36.428404 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:36.428410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:36.428538 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 14:51:36.428627 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:51:36.428759 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260928 14:51:36.429630 3832 heartbeater.cc:347] Connected to a master server at 127.18.96.190:41845
I20260928 14:51:36.429749 3832 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:36.429929 3832 heartbeater.cc:511] Master 127.18.96.190:41845 requested a full tablet report, sending...
I20260928 14:51:36.430344 3371 ts_manager.cc:194] Registered new tserver with Master: d00beb9b15cc439eacf86e55c0a180db (127.18.96.131:37649)
I20260928 14:51:36.430684 3371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:49669
I20260928 14:51:36.439349 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:51:36.447126 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.440931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.18.96.190:41845)
0928 14:51:36.441005 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.441007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.441136 (+ 129us) server_negotiation.cc:407] Connection header received
0928 14:51:36.441219 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.441219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.441230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.441301 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:36.441874 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.442385 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.443126 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.443321 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.443398 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.444637 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:36.444638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:36.444639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:36.444646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:36.445954 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:36.446292 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:36.446294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:36.446294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:36.446354 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:36.446568 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:36.446569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:36.446569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:36.446726 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:36.446833 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.446937 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.447021 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:51:36.448732 3371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:51:36.450430 3371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:51:36.457027 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.452842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:44967 (local address 127.0.0.1:33750)
0928 14:51:36.453004 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.453047 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:36.453113 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:36.453546 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:36.453547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:36.453551 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:36.453618 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.453620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.454638 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.454638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.455735 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.455737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.456714 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.456715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.456797 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.456812 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 14:51:36.456891 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:51:36.456940 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":16,"thread_start_us":97,"threads_started":1}
I20260928 14:51:36.457072 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.453172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.18.96.129:44967)
0928 14:51:36.453334 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.453338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.453350 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:51:36.453394 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.453396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.453413 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.453698 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:36.453785 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.454536 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.455826 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.456622 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.456701 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.456767 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.456930 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.456977 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260928 14:51:36.457140 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.452843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:37649 (local address 127.0.0.1:49516)
0928 14:51:36.453217 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.453226 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:36.453264 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:36.453491 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:36.453494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:36.453497 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:36.453570 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.453571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.454486 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.454487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.455945 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.455948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.456890 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.456891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.456956 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.456959 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:51:36.457017 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:51:36.457057 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":41,"thread_start_us":60,"threads_started":1}
I20260928 14:51:36.457414 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.453109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.18.96.131:37649)
0928 14:51:36.453313 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.453318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.453328 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:51:36.453369 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.453371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.453394 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.453482 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:36.453653 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.454388 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.456039 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.456724 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.456786 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.456830 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.457286 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.457331 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
I20260928 14:51:36.457870 3496 tablet_service.cc:1467] Processing CreateTablet for tablet 98aae86e15ba4f6787650eeb8de0b80a (DEFAULT_TABLE table=test-table [id=9c37b55fc02448239304e2b93ca0d72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:36.458016 3766 tablet_service.cc:1467] Processing CreateTablet for tablet 98aae86e15ba4f6787650eeb8de0b80a (DEFAULT_TABLE table=test-table [id=9c37b55fc02448239304e2b93ca0d72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:36.458256 3496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98aae86e15ba4f6787650eeb8de0b80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:36.458360 3766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98aae86e15ba4f6787650eeb8de0b80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:36.458869 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.452844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:43677 (local address 127.0.0.1:55902)
0928 14:51:36.453350 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.453361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:36.453413 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:36.454915 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:36.454916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:36.454918 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:36.454956 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.454957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.456202 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.456203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.457047 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.457049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.458647 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.458649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.458708 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.458712 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:36.458762 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:51:36.458801 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":7,"thread_start_us":95,"threads_started":1}
I20260928 14:51:36.459226 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.453128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.18.96.130:43677)
0928 14:51:36.454726 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.454731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.454756 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:51:36.454799 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.454801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.454817 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.454901 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:36.455028 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.456092 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.457743 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.458490 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.458559 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.458612 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.459099 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.459147 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1555,"thread_start_us":87,"threads_started":1}
I20260928 14:51:36.459919 3632 tablet_service.cc:1467] Processing CreateTablet for tablet 98aae86e15ba4f6787650eeb8de0b80a (DEFAULT_TABLE table=test-table [id=9c37b55fc02448239304e2b93ca0d72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:36.460268 3632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98aae86e15ba4f6787650eeb8de0b80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:36.461390 3850 tablet_bootstrap.cc:492] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a: Bootstrap starting.
I20260928 14:51:36.461861 3851 tablet_bootstrap.cc:492] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: Bootstrap starting.
I20260928 14:51:36.462183 3850 tablet_bootstrap.cc:654] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:36.462535 3851 tablet_bootstrap.cc:654] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:36.462710 3850 log.cc:824] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:36.463111 3851 log.cc:824] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:36.463508 3850 tablet_bootstrap.cc:492] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a: No bootstrap required, opened a new log
I20260928 14:51:36.463584 3850 ts_tablet_manager.cc:1397] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:51:36.463639 3854 tablet_bootstrap.cc:492] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: Bootstrap starting.
I20260928 14:51:36.464000 3851 tablet_bootstrap.cc:492] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: No bootstrap required, opened a new log
I20260928 14:51:36.464074 3851 ts_tablet_manager.cc:1397] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:51:36.464275 3854 tablet_bootstrap.cc:654] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:36.464841 3854 log.cc:824] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:36.465543 3854 tablet_bootstrap.cc:492] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: No bootstrap required, opened a new log
I20260928 14:51:36.465624 3854 ts_tablet_manager.cc:1397] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:51:36.465812 3850 raft_consensus.cc:348] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } }
I20260928 14:51:36.466025 3850 raft_consensus.cc:374] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:36.466074 3850 raft_consensus.cc:729] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d6a22901fb49e68649950618cc5e9a, State: Initialized, Role: FOLLOWER
I20260928 14:51:36.466096 3851 raft_consensus.cc:348] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } }
I20260928 14:51:36.466259 3851 raft_consensus.cc:374] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:36.466216 3850 consensus_queue.cc:260] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } }
I20260928 14:51:36.466298 3851 raft_consensus.cc:729] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00beb9b15cc439eacf86e55c0a180db, State: Initialized, Role: FOLLOWER
I20260928 14:51:36.466403 3851 consensus_queue.cc:260] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } }
I20260928 14:51:36.466553 3850 ts_tablet_manager.cc:1428] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:51:36.466599 3562 heartbeater.cc:503] Master 127.18.96.190:41845 was elected leader, sending a full tablet report...
I20260928 14:51:36.466668 3851 ts_tablet_manager.cc:1428] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:51:36.466728 3832 heartbeater.cc:503] Master 127.18.96.190:41845 was elected leader, sending a full tablet report...
I20260928 14:51:36.467515 3854 raft_consensus.cc:348] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } }
I20260928 14:51:36.467707 3854 raft_consensus.cc:374] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:36.467756 3854 raft_consensus.cc:729] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f177c3bb3e341049518c6de39e869f6, State: Initialized, Role: FOLLOWER
I20260928 14:51:36.467900 3854 consensus_queue.cc:260] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } }
I20260928 14:51:36.468212 3698 heartbeater.cc:503] Master 127.18.96.190:41845 was elected leader, sending a full tablet report...
I20260928 14:51:36.468214 3854 ts_tablet_manager.cc:1428] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260928 14:51:36.504860 3699 tablet.cc:2369] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:51:36.545077 3857 raft_consensus.cc:482] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:36.545279 3857 raft_consensus.cc:504] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } }
I20260928 14:51:36.545698 3857 leader_election.cc:288] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f177c3bb3e341049518c6de39e869f6 (127.18.96.130:43677), c9d6a22901fb49e68649950618cc5e9a (127.18.96.129:44967)
I20260928 14:51:36.548702 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.545917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:44967 (local address 127.18.96.131:57991)
0928 14:51:36.546002 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.546014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:36.546064 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:36.546237 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:36.546238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:36.546239 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:36.546325 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.546327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.547018 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.547019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.547782 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.547784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.548415 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.548416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.548503 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.548507 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:36.548575 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:51:36.548623 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:51:36.548702 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.546020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:57991 (local address 127.18.96.129:44967)
0928 14:51:36.546063 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.546065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.546075 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:51:36.546140 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.546141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.546158 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.546231 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:36.546405 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.546907 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.547915 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.548316 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.548407 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.548416 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.548581 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.548625 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:51:36.548969 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.545926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:43677 (local address 127.18.96.131:39123)
0928 14:51:36.546132 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.546142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:36.546238 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:36.546390 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:36.546391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:36.546392 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:36.546469 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.546471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.547217 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.547218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.548070 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.548072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.548647 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.548648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.548724 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.548728 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:36.548800 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:51:36.548850 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":2,"thread_start_us":130,"threads_started":1}
I20260928 14:51:36.549099 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.546003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:39123 (local address 127.18.96.130:43677)
0928 14:51:36.546053 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.546055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.546227 (+ 172us) server_negotiation.cc:407] Connection header received
0928 14:51:36.546306 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.546307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.546319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.546388 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:36.546599 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.547124 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.548169 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.548560 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.548647 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.548665 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.548953 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.549007 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:51:36.549108 3516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aae86e15ba4f6787650eeb8de0b80a" candidate_uuid: "d00beb9b15cc439eacf86e55c0a180db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d6a22901fb49e68649950618cc5e9a" is_pre_election: true
I20260928 14:51:36.549311 3516 raft_consensus.cc:2454] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00beb9b15cc439eacf86e55c0a180db in term 0.
I20260928 14:51:36.549409 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aae86e15ba4f6787650eeb8de0b80a" candidate_uuid: "d00beb9b15cc439eacf86e55c0a180db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f177c3bb3e341049518c6de39e869f6" is_pre_election: true
I20260928 14:51:36.549567 3652 raft_consensus.cc:2454] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00beb9b15cc439eacf86e55c0a180db in term 0.
I20260928 14:51:36.549600 3721 leader_election.cc:302] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9d6a22901fb49e68649950618cc5e9a, d00beb9b15cc439eacf86e55c0a180db; no voters:
I20260928 14:51:36.549860 3857 raft_consensus.cc:2781] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:51:36.549937 3857 raft_consensus.cc:482] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:51:36.549968 3857 raft_consensus.cc:3037] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:36.550763 3857 raft_consensus.cc:504] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } }
I20260928 14:51:36.550954 3857 leader_election.cc:288] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [CANDIDATE]: Term 1 election: Requested vote from peers 0f177c3bb3e341049518c6de39e869f6 (127.18.96.130:43677), c9d6a22901fb49e68649950618cc5e9a (127.18.96.129:44967)
I20260928 14:51:36.551208 3516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aae86e15ba4f6787650eeb8de0b80a" candidate_uuid: "d00beb9b15cc439eacf86e55c0a180db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d6a22901fb49e68649950618cc5e9a"
I20260928 14:51:36.551244 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aae86e15ba4f6787650eeb8de0b80a" candidate_uuid: "d00beb9b15cc439eacf86e55c0a180db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f177c3bb3e341049518c6de39e869f6"
I20260928 14:51:36.551342 3516 raft_consensus.cc:3037] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:36.551357 3652 raft_consensus.cc:3037] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:36.552256 3516 raft_consensus.cc:2454] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00beb9b15cc439eacf86e55c0a180db in term 1.
I20260928 14:51:36.552264 3652 raft_consensus.cc:2454] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00beb9b15cc439eacf86e55c0a180db in term 1.
I20260928 14:51:36.552469 3722 leader_election.cc:302] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f177c3bb3e341049518c6de39e869f6, d00beb9b15cc439eacf86e55c0a180db; no voters:
I20260928 14:51:36.552580 3857 raft_consensus.cc:2781] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:36.552729 3857 raft_consensus.cc:686] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 1 LEADER]: Becoming Leader. State: Replica: d00beb9b15cc439eacf86e55c0a180db, State: Running, Role: LEADER
I20260928 14:51:36.552835 3857 consensus_queue.cc:237] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } }
I20260928 14:51:36.553862 3370 catalog_manager.cc:5495] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db reported cstate change: term changed from 0 to 1, leader changed from <none> to d00beb9b15cc439eacf86e55c0a180db (127.18.96.131). New cstate: current_term: 1 leader_uuid: "d00beb9b15cc439eacf86e55c0a180db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00beb9b15cc439eacf86e55c0a180db" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 37649 } health_report { overall_health: HEALTHY } } }
I20260928 14:51:36.578202 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.575489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.18.96.131:37649)
0928 14:51:36.575567 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.575570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.575581 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:51:36.575667 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.575668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.575689 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.575755 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:51:36.576062 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.576535 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.577411 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.577608 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.577663 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.577853 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:51:36.577955 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:51:36.578018 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.578101 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.578119 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:51:36.580183 3652 raft_consensus.cc:1261] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [term 1 FOLLOWER]: Refusing update from remote peer d00beb9b15cc439eacf86e55c0a180db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:36.580263 3516 raft_consensus.cc:1261] T 98aae86e15ba4f6787650eeb8de0b80a P c9d6a22901fb49e68649950618cc5e9a [term 1 FOLLOWER]: Refusing update from remote peer d00beb9b15cc439eacf86e55c0a180db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:36.580593 3860 consensus_queue.cc:1035] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f177c3bb3e341049518c6de39e869f6" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:36.580760 3857 consensus_queue.cc:1035] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9d6a22901fb49e68649950618cc5e9a" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:36.583333 3864 mvcc.cc:205] Tried to move back new op lower bound from 7334326667589505024 to 7334326667481161728. Current Snapshot: MvccSnapshot[applied={T|T < 7334326667589505024}]
I20260928 14:51:36.584388 3865 mvcc.cc:205] Tried to move back new op lower bound from 7334326667589505024 to 7334326667481161728. Current Snapshot: MvccSnapshot[applied={T|T < 7334326667589505024}]
I20260928 14:51:36.585807 3371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:36.585912 3371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:36.586850 3371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ba348c9f54425abe2c8b56fa72bb3c: Sending DeleteTablet for 3 replicas of tablet 98aae86e15ba4f6787650eeb8de0b80a
I20260928 14:51:36.587240 3632 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aae86e15ba4f6787650eeb8de0b80a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:36 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:55902
I20260928 14:51:36.587240 3766 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aae86e15ba4f6787650eeb8de0b80a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:36 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:49516
I20260928 14:51:36.587571 3872 tablet_replica.cc:330] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: stopping tablet replica
I20260928 14:51:36.587585 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 3433
I20260928 14:51:36.587595 3496 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aae86e15ba4f6787650eeb8de0b80a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:36 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:33750
I20260928 14:51:36.587783 3872 raft_consensus.cc:2229] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:51:36.587984 3872 raft_consensus.cc:2258] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:36.588171 3873 tablet_replica.cc:330] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: stopping tablet replica
I20260928 14:51:36.588272 3873 raft_consensus.cc:2229] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:51:36.588384 3873 raft_consensus.cc:2258] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:36.588855 3873 ts_tablet_manager.cc:1905] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:36.589673 3872 ts_tablet_manager.cc:1905] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:36.590605 3873 ts_tablet_manager.cc:1918] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:36.590679 3873 log.cc:1196] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/98aae86e15ba4f6787650eeb8de0b80a
I20260928 14:51:36.590981 3873 ts_tablet_manager.cc:1939] T 98aae86e15ba4f6787650eeb8de0b80a P 0f177c3bb3e341049518c6de39e869f6: Deleting consensus metadata
I20260928 14:51:36.591420 3358 catalog_manager.cc:4841] TS 0f177c3bb3e341049518c6de39e869f6 (127.18.96.130:43677): tablet 98aae86e15ba4f6787650eeb8de0b80a (table test-table [id=9c37b55fc02448239304e2b93ca0d72e]) successfully deleted
I20260928 14:51:36.591924 3872 ts_tablet_manager.cc:1918] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:36.592074 3872 log.cc:1196] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/98aae86e15ba4f6787650eeb8de0b80a
I20260928 14:51:36.592422 3872 ts_tablet_manager.cc:1939] T 98aae86e15ba4f6787650eeb8de0b80a P d00beb9b15cc439eacf86e55c0a180db: Deleting consensus metadata
I20260928 14:51:36.592868 3356 catalog_manager.cc:4841] TS d00beb9b15cc439eacf86e55c0a180db (127.18.96.131:37649): tablet 98aae86e15ba4f6787650eeb8de0b80a (table test-table [id=9c37b55fc02448239304e2b93ca0d72e]) successfully deleted
W20260928 14:51:36.595263 3357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:44967 (error 108)
I20260928 14:51:36.595496 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 3568
I20260928 14:51:36.595819 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.595694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:44967 (local address 127.0.0.1:33758)
0928 14:51:36.595746 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.595778 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:51:36.596060 3357 catalog_manager.cc:4584] TS c9d6a22901fb49e68649950618cc5e9a (127.18.96.129:44967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98aae86e15ba4f6787650eeb8de0b80a: Network error: Client connection negotiation failed: client connection to 127.18.96.129:44967: connect: Connection refused (error 111)
I20260928 14:51:36.604203 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 3703
I20260928 14:51:36.609817 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 3338
2026-09-28T14:51:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2065 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T14:51:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:36Z Disabled control of system clock
I20260928 14:51:36.638026 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34843
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:43659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:51:36.719594 3878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:36.719834 3878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:36.719856 3878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:36.721074 3878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:36.721122 3878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:36.721133 3878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:36.721146 3878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:36.721158 3878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:36.722925 3878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:34843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:36.723170 3878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:36.723451 3878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:36.728621 3883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.728641 3884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.728645 3886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:36.728878 3878 server_base.cc:1034] running on GCE node
I20260928 14:51:36.729176 3878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:36.729501 3878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:36.730691 3878 hybrid_clock.cc:648] HybridClock initialized: now 1790607096730655 us; error 49 us; skew 500 ppm
I20260928 14:51:36.732350 3878 webserver.cc:460] Webserver started at http://127.18.96.190:38573/ using document root <none> and password file <none>
I20260928 14:51:36.732637 3878 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:36.732687 3878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:36.732828 3878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:36.734047 3878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "65eb64533cea4567bed320bcd2390cd0"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "701c432eb42183e47286f988bc9cdc37"
server_key_iv: "53547a41149aa5f46f82b53ede26794f"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.734439 3878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "65eb64533cea4567bed320bcd2390cd0"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "701c432eb42183e47286f988bc9cdc37"
server_key_iv: "53547a41149aa5f46f82b53ede26794f"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.735934 3878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:51:36.736815 3893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.737088 3878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:36.737274 3878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "65eb64533cea4567bed320bcd2390cd0"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "701c432eb42183e47286f988bc9cdc37"
server_key_iv: "53547a41149aa5f46f82b53ede26794f"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.737365 3878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:36.748734 3878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:36.749073 3878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:36.749238 3878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:36.753458 3878 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:34843
I20260928 14:51:36.753510 3945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:34843 every 8 connection(s)
I20260928 14:51:36.753896 3878 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:36.754891 3946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:36.757395 3946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0: Bootstrap starting.
I20260928 14:51:36.758008 3946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:36.758328 3946 log.cc:824] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:36.759087 3946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0: No bootstrap required, opened a new log
I20260928 14:51:36.760869 3946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65eb64533cea4567bed320bcd2390cd0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34843 } }
I20260928 14:51:36.761020 3946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:36.761044 3946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65eb64533cea4567bed320bcd2390cd0, State: Initialized, Role: FOLLOWER
I20260928 14:51:36.761188 3946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65eb64533cea4567bed320bcd2390cd0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34843 } }
I20260928 14:51:36.761269 3946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:36.761315 3946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:36.761368 3946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:36.761922 3946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65eb64533cea4567bed320bcd2390cd0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34843 } }
I20260928 14:51:36.762058 3946 leader_election.cc:302] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65eb64533cea4567bed320bcd2390cd0; no voters:
I20260928 14:51:36.762261 3946 leader_election.cc:288] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:36.762362 3949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:36.762534 3946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:36.762707 3949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [term 1 LEADER]: Becoming Leader. State: Replica: 65eb64533cea4567bed320bcd2390cd0, State: Running, Role: LEADER
I20260928 14:51:36.762768 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 3878
I20260928 14:51:36.762868 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:36.762825 3949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65eb64533cea4567bed320bcd2390cd0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34843 } }
I20260928 14:51:36.763083 18818 external_mini_cluster.cc:1428] Setting key 5a3669049e0ba9ce58acd3a296b6f61d
I20260928 14:51:36.763329 3951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65eb64533cea4567bed320bcd2390cd0. Latest consensus state: current_term: 1 leader_uuid: "65eb64533cea4567bed320bcd2390cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65eb64533cea4567bed320bcd2390cd0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34843 } } }
I20260928 14:51:36.763790 3951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:36.763710 3950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65eb64533cea4567bed320bcd2390cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65eb64533cea4567bed320bcd2390cd0" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34843 } } }
I20260928 14:51:36.764219 3950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:36.764075 3966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:36.764940 3966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:36.766489 3966 catalog_manager.cc:1347] Generated new cluster ID: ee4db86f60124bccbc4ee00dc9407784
I20260928 14:51:36.766546 3966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:36.769485 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.763806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.18.96.190:34843)
0928 14:51:36.764095 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.764105 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.764219 (+ 114us) server_negotiation.cc:407] Connection header received
0928 14:51:36.764343 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.764350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.764420 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.764560 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:36.764815 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.766208 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.767309 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.767964 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.768095 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.768743 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:36.768747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:36.768747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:36.768961 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:36.769004 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:36.769019 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:36.769090 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.769165 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.769249 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":127,"threads_started":1}
I20260928 14:51:36.780386 3966 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:36.780962 3966 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:36.786208 3966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0: Generated new TSK 0
I20260928 14:51:36.786424 3966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:36.793113 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34843
--builtin_ntp_servers=127.18.96.148:43659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:51:36.878151 3970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:36.878357 3970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:36.878381 3970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:36.879601 3970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:36.879647 3970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:36.879673 3970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:51:36.881064 3970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:36.881346 3970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:36.881732 3970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:36.887141 3978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.887147 3975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:36.887347 3976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:36.887735 3970 server_base.cc:1034] running on GCE node
I20260928 14:51:36.887989 3970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:36.888253 3970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:36.889454 3970 hybrid_clock.cc:648] HybridClock initialized: now 1790607096889438 us; error 29 us; skew 500 ppm
I20260928 14:51:36.890910 3970 webserver.cc:460] Webserver started at http://127.18.96.129:40913/ using document root <none> and password file <none>
I20260928 14:51:36.891247 3970 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:36.891328 3970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:36.891484 3970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:36.892707 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "086d55307b4c47c9a945ba96cc85420d"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "21a35ea4b1312d8ff7726ee493fc03a2"
server_key_iv: "64219c4560f94b0608bac0f478abf6d7"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.893095 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "086d55307b4c47c9a945ba96cc85420d"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "21a35ea4b1312d8ff7726ee493fc03a2"
server_key_iv: "64219c4560f94b0608bac0f478abf6d7"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.894582 3970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:51:36.895442 3984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.895644 3970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:36.895763 3970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "086d55307b4c47c9a945ba96cc85420d"
format_stamp: "Formatted at 2026-09-28 14:51:36 on dist-test-slave-j968"
server_key: "21a35ea4b1312d8ff7726ee493fc03a2"
server_key_iv: "64219c4560f94b0608bac0f478abf6d7"
server_key_version: "encryptionkey@0"
I20260928 14:51:36.895843 3970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:36.906599 3970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:36.906906 3970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:36.907025 3970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:36.907246 3970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:36.907708 3970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:36.907745 3970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.907768 3970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:36.907819 3970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:36.913825 3970 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:38993
I20260928 14:51:36.913925 4098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:38993 every 8 connection(s)
I20260928 14:51:36.914263 3970 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:51:36.918598 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 3970
I20260928 14:51:36.918684 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:51:36.918900 18818 external_mini_cluster.cc:1428] Setting key 0b89748e9b1b07a5dd5844ceb9d62988
I20260928 14:51:36.920076 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34843
--builtin_ntp_servers=127.18.96.148:43659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:51:36.920445 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.915061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:35049 (local address 127.18.96.190:34843)
0928 14:51:36.915169 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:36.915172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:36.915381 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:51:36.916040 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:36.916041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:36.916058 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:36.916110 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:36.916825 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.917383 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.918433 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.918680 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.918864 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.919928 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:36.919929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:36.919930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:36.919934 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:36.919950 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:36.920031 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:36.920083 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:36.920285 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 14:51:36.920328 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:51:36.920955 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:36.915046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34843 (local address 127.18.96.129:35049)
0928 14:51:36.915256 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:36.915293 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:36.915915 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:36.916181 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:36.916186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:36.916196 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:36.916728 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.916733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.917494 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.917495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.918326 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:36.918328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:36.918850 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:36.918852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:36.919050 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:36.919715 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:51:36.919716 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:36.919771 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:51:36.919801 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:36.920162 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:36.920171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:36.920174 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:51:36.920562 (+ 388us) client_negotiation.cc:240] Negotiation successful
0928 14:51:36.920691 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260928 14:51:36.921634 4099 heartbeater.cc:347] Connected to a master server at 127.18.96.190:34843
I20260928 14:51:36.921720 4099 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:36.921918 4099 heartbeater.cc:511] Master 127.18.96.190:34843 requested a full tablet report, sending...
I20260928 14:51:36.922528 3910 ts_manager.cc:194] Registered new tserver with Master: 086d55307b4c47c9a945ba96cc85420d (127.18.96.129:38993)
I20260928 14:51:36.923158 3910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.129:35049
W20260928 14:51:37.004491 4102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:37.004717 4102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:37.004740 4102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:37.005879 4102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:37.005913 4102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:37.005939 4102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:51:37.007232 4102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:37.007431 4102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:37.007699 4102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:37.013135 4110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:37.013146 4107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:37.013146 4108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:37.013311 4102 server_base.cc:1034] running on GCE node
I20260928 14:51:37.013554 4102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:37.013849 4102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:37.015003 4102 hybrid_clock.cc:648] HybridClock initialized: now 1790607097014971 us; error 45 us; skew 500 ppm
I20260928 14:51:37.016485 4102 webserver.cc:460] Webserver started at http://127.18.96.130:42393/ using document root <none> and password file <none>
I20260928 14:51:37.016788 4102 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:37.016839 4102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:37.016963 4102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:37.018179 4102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "cb3e9f0cdb9545849155d4dc96f8aecf"
format_stamp: "Formatted at 2026-09-28 14:51:37 on dist-test-slave-j968"
server_key: "caba841b089be00d0f522807626b9125"
server_key_iv: "b373c5528e4977856f1ed83a7ed397d2"
server_key_version: "encryptionkey@0"
I20260928 14:51:37.018541 4102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "cb3e9f0cdb9545849155d4dc96f8aecf"
format_stamp: "Formatted at 2026-09-28 14:51:37 on dist-test-slave-j968"
server_key: "caba841b089be00d0f522807626b9125"
server_key_iv: "b373c5528e4977856f1ed83a7ed397d2"
server_key_version: "encryptionkey@0"
I20260928 14:51:37.019902 4102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 14:51:37.020742 4116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:37.020918 4102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:51:37.021003 4102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "cb3e9f0cdb9545849155d4dc96f8aecf"
format_stamp: "Formatted at 2026-09-28 14:51:37 on dist-test-slave-j968"
server_key: "caba841b089be00d0f522807626b9125"
server_key_iv: "b373c5528e4977856f1ed83a7ed397d2"
server_key_version: "encryptionkey@0"
I20260928 14:51:37.021126 4102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:37.049336 4102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:37.049656 4102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:37.049770 4102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:37.049973 4102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:37.050468 4102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:37.050505 4102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:37.050530 4102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:37.050642 4102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:37.056671 4102 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:34061
I20260928 14:51:37.056716 4230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:34061 every 8 connection(s)
I20260928 14:51:37.057066 4102 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:51:37.062850 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.057865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:49681 (local address 127.18.96.190:34843)
0928 14:51:37.057947 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:37.057949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:37.058219 (+ 270us) server_negotiation.cc:407] Connection header received
0928 14:51:37.058833 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:37.058835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:37.058848 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:37.058901 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:37.059678 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.060284 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.061312 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.061523 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.061613 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.062537 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:37.062539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:37.062539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:37.062545 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:37.062562 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:37.062567 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:37.062620 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:37.062725 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 14:51:37.062763 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:51:37.063117 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.057859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34843 (local address 127.18.96.130:49681)
0928 14:51:37.058066 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:37.058119 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:37.058712 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:37.058992 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:37.058996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:37.059005 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:37.059570 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.059575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.060391 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.060392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.061211 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.061213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.061627 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.061628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.061765 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.062387 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:51:37.062387 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:37.062418 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:51:37.062442 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:37.062623 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:37.062629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:37.062633 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:37.062716 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:51:37.062832 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260928 14:51:37.063702 4231 heartbeater.cc:347] Connected to a master server at 127.18.96.190:34843
I20260928 14:51:37.063831 4231 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:37.064045 4231 heartbeater.cc:511] Master 127.18.96.190:34843 requested a full tablet report, sending...
I20260928 14:51:37.064532 3910 ts_manager.cc:194] Registered new tserver with Master: cb3e9f0cdb9545849155d4dc96f8aecf (127.18.96.130:34061)
I20260928 14:51:37.064898 3910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.130:49681
I20260928 14:51:37.066834 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 4102
I20260928 14:51:37.066962 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:51:37.067164 18818 external_mini_cluster.cc:1428] Setting key e090ae3122b1ca272578022d4841bb0f
I20260928 14:51:37.068096 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34843
--builtin_ntp_servers=127.18.96.148:43659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:51:37.155769 4234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:37.156111 4234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:37.156147 4234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:37.157634 4234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:37.157685 4234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:37.157716 4234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:51:37.159062 4234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:34843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:37.159260 4234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:37.159521 4234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:37.165126 4239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:37.165187 4234 server_base.cc:1034] running on GCE node
W20260928 14:51:37.165190 4240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:37.165167 4242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:37.165591 4234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:37.165818 4234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:37.166960 4234 hybrid_clock.cc:648] HybridClock initialized: now 1790607097166904 us; error 71 us; skew 500 ppm
I20260928 14:51:37.168273 4234 webserver.cc:460] Webserver started at http://127.18.96.131:41009/ using document root <none> and password file <none>
I20260928 14:51:37.168471 4234 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:37.168516 4234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:37.168655 4234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:37.169677 4234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "25b78ab1dc1647ddabc26b45183c0bbb"
format_stamp: "Formatted at 2026-09-28 14:51:37 on dist-test-slave-j968"
server_key: "b14585232f7aa7465e8e8413a7f40aa8"
server_key_iv: "255199e9ede3cbd0024258018cb292be"
server_key_version: "encryptionkey@0"
I20260928 14:51:37.169971 4234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "25b78ab1dc1647ddabc26b45183c0bbb"
format_stamp: "Formatted at 2026-09-28 14:51:37 on dist-test-slave-j968"
server_key: "b14585232f7aa7465e8e8413a7f40aa8"
server_key_iv: "255199e9ede3cbd0024258018cb292be"
server_key_version: "encryptionkey@0"
I20260928 14:51:37.171213 4234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:51:37.171941 4248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:37.172145 4234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:37.172237 4234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "25b78ab1dc1647ddabc26b45183c0bbb"
format_stamp: "Formatted at 2026-09-28 14:51:37 on dist-test-slave-j968"
server_key: "b14585232f7aa7465e8e8413a7f40aa8"
server_key_iv: "255199e9ede3cbd0024258018cb292be"
server_key_version: "encryptionkey@0"
I20260928 14:51:37.172298 4234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:37.178948 4234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:37.179229 4234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:37.179339 4234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:37.179529 4234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:37.179858 4234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:37.179955 4234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:37.179978 4234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:37.179991 4234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:37.185688 4234 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:39505
I20260928 14:51:37.185837 4361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:39505 every 8 connection(s)
I20260928 14:51:37.186122 4234 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:51:37.191278 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.186866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:50555 (local address 127.18.96.190:34843)
0928 14:51:37.186927 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:37.186929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:37.187198 (+ 269us) server_negotiation.cc:407] Connection header received
0928 14:51:37.187681 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:37.187682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:37.187693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:37.187748 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:37.188453 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.188958 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.189953 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.190156 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.190217 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.191033 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:37.191035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:37.191035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:37.191040 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:37.191055 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:37.191059 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:37.191104 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:37.191181 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 14:51:37.191212 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:51:37.191466 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.186863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:34843 (local address 127.18.96.131:50555)
0928 14:51:37.187045 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:37.187105 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:37.187598 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:37.187810 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:37.187813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:37.187822 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:37.188373 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.188379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.189049 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.189050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.189870 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.189872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.190240 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.190241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.190362 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.190906 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:51:37.190906 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:37.190935 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:51:37.190956 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:37.191112 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:37.191117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:37.191119 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:51:37.191172 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:51:37.191256 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 14:51:37.192149 4362 heartbeater.cc:347] Connected to a master server at 127.18.96.190:34843
I20260928 14:51:37.192262 4362 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:37.192449 4362 heartbeater.cc:511] Master 127.18.96.190:34843 requested a full tablet report, sending...
I20260928 14:51:37.192929 3910 ts_manager.cc:194] Registered new tserver with Master: 25b78ab1dc1647ddabc26b45183c0bbb (127.18.96.131:39505)
I20260928 14:51:37.193320 3910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.131:50555
I20260928 14:51:37.193337 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 4234
I20260928 14:51:37.193441 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:51:37.193696 18818 external_mini_cluster.cc:1428] Setting key 9b6faf0905508d6c74a4ae398dde2082
I20260928 14:51:37.194766 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:51:37.199678 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.196317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.18.96.190:34843)
0928 14:51:37.196398 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:37.196400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:37.196419 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:51:37.196480 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:37.196481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:37.196493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:37.196641 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:37.196818 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.197369 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.198291 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.198523 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.198595 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.199452 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:37.199454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:37.199454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:37.199458 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:37.199473 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:37.199477 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:37.199526 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:37.199586 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 14:51:37.199612 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:51:37.201488 3910 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:51:37.203426 3910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:51:37.210441 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.206410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.18.96.129:38993)
0928 14:51:37.206577 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:37.206581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:37.206783 (+ 202us) server_negotiation.cc:407] Connection header received
0928 14:51:37.206831 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:37.206834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:37.206851 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:37.206924 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:37.207116 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.207926 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.209339 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.209946 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.210120 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.210233 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:37.210308 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 14:51:37.210350 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260928 14:51:37.210429 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.206300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38993 (local address 127.0.0.1:46038)
0928 14:51:37.206658 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:37.206666 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:37.206727 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:37.206999 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:37.207000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:37.207002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:37.207054 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.207055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.208007 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.208008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.209235 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.209238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.210122 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.210123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.210208 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.210222 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:51:37.210295 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:51:37.210350 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":73,"threads_started":1}
I20260928 14:51:37.210958 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.206064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:34061 (local address 127.0.0.1:58150)
0928 14:51:37.206430 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:37.206475 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:37.206551 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:37.206874 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:37.206877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:37.206879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:37.206967 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.206970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.208357 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.208358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.209410 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.209412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.210712 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.210713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.210781 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.210785 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:37.210844 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:51:37.210889 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260928 14:51:37.211006 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.206211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.18.96.130:34061)
0928 14:51:37.206431 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:37.206437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:37.206643 (+ 206us) server_negotiation.cc:407] Connection header received
0928 14:51:37.206694 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:37.206696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:37.206713 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:37.206788 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:37.207134 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.208276 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.209489 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.210235 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.210308 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.210391 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:37.210871 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 14:51:37.210916 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260928 14:51:37.211357 4032 tablet_service.cc:1467] Processing CreateTablet for tablet 552fc69d52c1458aaa336cf74ee7c75d (DEFAULT_TABLE table=test-table [id=c705a7e058de40afb3f51bb04008cbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:37.211717 4032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552fc69d52c1458aaa336cf74ee7c75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:37.211763 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.206069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:39505 (local address 127.0.0.1:54874)
0928 14:51:37.206431 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:37.206472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:37.206571 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:37.206874 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:37.206877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:37.206881 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:37.206967 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.206971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.209508 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.209508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.210571 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.210574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.211518 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.211519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.211581 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.211584 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:51:37.211634 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:51:37.211674 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":7,"thread_start_us":95,"threads_started":1}
I20260928 14:51:37.211772 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.206187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.18.96.131:39505)
0928 14:51:37.206640 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:37.206644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:37.206655 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:51:37.206714 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:37.206716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:37.206731 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:37.206802 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:37.207056 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.208269 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.210666 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.211410 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.211508 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.211590 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:37.211646 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 14:51:37.211683 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":106,"threads_started":1}
I20260928 14:51:37.212391 4296 tablet_service.cc:1467] Processing CreateTablet for tablet 552fc69d52c1458aaa336cf74ee7c75d (DEFAULT_TABLE table=test-table [id=c705a7e058de40afb3f51bb04008cbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:37.212713 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552fc69d52c1458aaa336cf74ee7c75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:37.214560 4376 tablet_bootstrap.cc:492] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d: Bootstrap starting.
I20260928 14:51:37.215257 4377 tablet_bootstrap.cc:492] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: Bootstrap starting.
I20260928 14:51:37.215322 4376 tablet_bootstrap.cc:654] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:37.215654 4376 log.cc:824] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:37.215716 4165 tablet_service.cc:1467] Processing CreateTablet for tablet 552fc69d52c1458aaa336cf74ee7c75d (DEFAULT_TABLE table=test-table [id=c705a7e058de40afb3f51bb04008cbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:37.216054 4377 tablet_bootstrap.cc:654] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:37.216106 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552fc69d52c1458aaa336cf74ee7c75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:37.216372 4377 log.cc:824] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:37.217131 4376 tablet_bootstrap.cc:492] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d: No bootstrap required, opened a new log
I20260928 14:51:37.217190 4377 tablet_bootstrap.cc:492] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: No bootstrap required, opened a new log
I20260928 14:51:37.217217 4376 ts_tablet_manager.cc:1397] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 14:51:37.217294 4377 ts_tablet_manager.cc:1397] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:51:37.219411 4376 raft_consensus.cc:348] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } }
I20260928 14:51:37.219581 4376 raft_consensus.cc:374] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:37.219625 4376 raft_consensus.cc:729] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086d55307b4c47c9a945ba96cc85420d, State: Initialized, Role: FOLLOWER
I20260928 14:51:37.219743 4376 consensus_queue.cc:260] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } }
I20260928 14:51:37.220041 4376 ts_tablet_manager.cc:1428] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:51:37.220052 4380 tablet_bootstrap.cc:492] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: Bootstrap starting.
I20260928 14:51:37.220176 4099 heartbeater.cc:503] Master 127.18.96.190:34843 was elected leader, sending a full tablet report...
I20260928 14:51:37.220346 4377 raft_consensus.cc:348] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } }
I20260928 14:51:37.220518 4377 raft_consensus.cc:374] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:37.220561 4377 raft_consensus.cc:729] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b78ab1dc1647ddabc26b45183c0bbb, State: Initialized, Role: FOLLOWER
I20260928 14:51:37.220679 4377 consensus_queue.cc:260] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } }
I20260928 14:51:37.220862 4380 tablet_bootstrap.cc:654] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:37.220939 4377 ts_tablet_manager.cc:1428] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 14:51:37.221064 4362 heartbeater.cc:503] Master 127.18.96.190:34843 was elected leader, sending a full tablet report...
I20260928 14:51:37.221215 4380 log.cc:824] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:37.222280 4380 tablet_bootstrap.cc:492] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: No bootstrap required, opened a new log
I20260928 14:51:37.222371 4380 ts_tablet_manager.cc:1397] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:51:37.224303 4380 raft_consensus.cc:348] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } }
I20260928 14:51:37.224447 4380 raft_consensus.cc:374] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:37.224493 4380 raft_consensus.cc:729] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3e9f0cdb9545849155d4dc96f8aecf, State: Initialized, Role: FOLLOWER
I20260928 14:51:37.224609 4380 consensus_queue.cc:260] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } }
I20260928 14:51:37.224853 4380 ts_tablet_manager.cc:1428] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 14:51:37.224957 4231 heartbeater.cc:503] Master 127.18.96.190:34843 was elected leader, sending a full tablet report...
W20260928 14:51:37.307669 4232 tablet.cc:2369] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:51:37.402577 4381 raft_consensus.cc:482] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:37.402756 4381 raft_consensus.cc:504] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } }
I20260928 14:51:37.403187 4381 leader_election.cc:288] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb3e9f0cdb9545849155d4dc96f8aecf (127.18.96.130:34061), 25b78ab1dc1647ddabc26b45183c0bbb (127.18.96.131:39505)
I20260928 14:51:37.407733 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.403480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:51381 (local address 127.18.96.131:39505)
0928 14:51:37.403531 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:37.403533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:37.403549 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:51:37.403633 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:37.403634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:37.403646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:37.403738 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:37.403955 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.404766 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.406730 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.407331 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.407396 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.407404 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:37.407623 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 14:51:37.407663 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:51:37.407749 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.403548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:39227 (local address 127.18.96.130:34061)
0928 14:51:37.403611 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:37.403613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:37.403728 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:51:37.403785 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:37.403786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:37.403802 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:37.403845 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:37.404041 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.404766 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.406730 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.407299 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.407386 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.407397 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:37.407612 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 14:51:37.407660 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:51:37.407774 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.403402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:39505 (local address 127.18.96.129:51381)
0928 14:51:37.403454 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:37.403467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:37.403564 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:37.403745 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:37.403746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:37.403747 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:37.403826 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.403842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.404895 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.404896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.406603 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.406607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.407417 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.407418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.407506 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.407514 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:51:37.407610 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:51:37.407649 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":208}
I20260928 14:51:37.407781 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.403478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:34061 (local address 127.18.96.129:39227)
0928 14:51:37.403630 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:37.403640 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:37.403728 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:37.403884 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:37.403885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:37.403887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:37.403964 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.403965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.404895 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.404896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.406603 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:37.406607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.407431 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.407431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:37.407506 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.407514 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:51:37.407585 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:51:37.407646 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":80,"thread_start_us":63,"threads_started":1}
I20260928 14:51:37.408237 4316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552fc69d52c1458aaa336cf74ee7c75d" candidate_uuid: "086d55307b4c47c9a945ba96cc85420d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" is_pre_election: true
I20260928 14:51:37.408373 4185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552fc69d52c1458aaa336cf74ee7c75d" candidate_uuid: "086d55307b4c47c9a945ba96cc85420d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" is_pre_election: true
I20260928 14:51:37.408432 4316 raft_consensus.cc:2454] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086d55307b4c47c9a945ba96cc85420d in term 0.
I20260928 14:51:37.408525 4185 raft_consensus.cc:2454] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086d55307b4c47c9a945ba96cc85420d in term 0.
I20260928 14:51:37.408694 3988 leader_election.cc:302] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086d55307b4c47c9a945ba96cc85420d, 25b78ab1dc1647ddabc26b45183c0bbb; no voters:
I20260928 14:51:37.408847 4381 raft_consensus.cc:2781] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:51:37.408907 4381 raft_consensus.cc:482] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:51:37.408936 4381 raft_consensus.cc:3037] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:37.409736 4381 raft_consensus.cc:504] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } }
I20260928 14:51:37.409902 4381 leader_election.cc:288] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [CANDIDATE]: Term 1 election: Requested vote from peers cb3e9f0cdb9545849155d4dc96f8aecf (127.18.96.130:34061), 25b78ab1dc1647ddabc26b45183c0bbb (127.18.96.131:39505)
I20260928 14:51:37.410054 4185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552fc69d52c1458aaa336cf74ee7c75d" candidate_uuid: "086d55307b4c47c9a945ba96cc85420d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf"
I20260928 14:51:37.410056 4316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552fc69d52c1458aaa336cf74ee7c75d" candidate_uuid: "086d55307b4c47c9a945ba96cc85420d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b78ab1dc1647ddabc26b45183c0bbb"
I20260928 14:51:37.410142 4185 raft_consensus.cc:3037] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:37.410141 4316 raft_consensus.cc:3037] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:37.411139 4316 raft_consensus.cc:2454] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086d55307b4c47c9a945ba96cc85420d in term 1.
I20260928 14:51:37.411139 4185 raft_consensus.cc:2454] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086d55307b4c47c9a945ba96cc85420d in term 1.
I20260928 14:51:37.411396 3985 leader_election.cc:302] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086d55307b4c47c9a945ba96cc85420d, cb3e9f0cdb9545849155d4dc96f8aecf; no voters:
I20260928 14:51:37.411501 4381 raft_consensus.cc:2781] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:37.411700 4381 raft_consensus.cc:686] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [term 1 LEADER]: Becoming Leader. State: Replica: 086d55307b4c47c9a945ba96cc85420d, State: Running, Role: LEADER
I20260928 14:51:37.411820 4381 consensus_queue.cc:237] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } }
I20260928 14:51:37.412802 3908 catalog_manager.cc:5495] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d reported cstate change: term changed from 0 to 1, leader changed from <none> to 086d55307b4c47c9a945ba96cc85420d (127.18.96.129). New cstate: current_term: 1 leader_uuid: "086d55307b4c47c9a945ba96cc85420d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "086d55307b4c47c9a945ba96cc85420d" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 38993 } health_report { overall_health: HEALTHY } } }
W20260928 14:51:37.414924 4100 tablet.cc:2369] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:51:37.436707 4363 tablet.cc:2369] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:51:37.442269 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.439654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.18.96.129:38993)
0928 14:51:37.439733 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:37.439735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:37.439745 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:51:37.439798 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:37.439799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:37.439821 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:37.439954 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:51:37.440110 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.440605 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.441488 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:37.441707 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:37.441773 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:37.441902 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:51:37.442015 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:51:37.442088 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:37.442106 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 14:51:37.442129 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:51:37.444283 4316 raft_consensus.cc:1261] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [term 1 FOLLOWER]: Refusing update from remote peer 086d55307b4c47c9a945ba96cc85420d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:37.444319 4185 raft_consensus.cc:1261] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [term 1 FOLLOWER]: Refusing update from remote peer 086d55307b4c47c9a945ba96cc85420d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:37.444664 4381 consensus_queue.cc:1035] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b78ab1dc1647ddabc26b45183c0bbb" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:37.444839 4386 consensus_queue.cc:1035] T 552fc69d52c1458aaa336cf74ee7c75d P 086d55307b4c47c9a945ba96cc85420d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb3e9f0cdb9545849155d4dc96f8aecf" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:37.447710 4390 mvcc.cc:205] Tried to move back new op lower bound from 7334326671128461312 to 7334326670999846912. Current Snapshot: MvccSnapshot[applied={T|T < 7334326671128461312}]
I20260928 14:51:37.448750 4391 mvcc.cc:205] Tried to move back new op lower bound from 7334326671128461312 to 7334326670999846912. Current Snapshot: MvccSnapshot[applied={T|T < 7334326671128461312}]
I20260928 14:51:37.450026 3909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:37.450124 3909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:37.450917 3909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65eb64533cea4567bed320bcd2390cd0: Sending DeleteTablet for 3 replicas of tablet 552fc69d52c1458aaa336cf74ee7c75d
I20260928 14:51:37.451264 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet 552fc69d52c1458aaa336cf74ee7c75d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:37 UTC) from {username='slave'} at 127.0.0.1:54874
I20260928 14:51:37.451318 4165 tablet_service.cc:1514] Processing DeleteTablet for tablet 552fc69d52c1458aaa336cf74ee7c75d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:37 UTC) from {username='slave'} at 127.0.0.1:58150
I20260928 14:51:37.451488 4398 tablet_replica.cc:330] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: stopping tablet replica
I20260928 14:51:37.451596 4399 tablet_replica.cc:330] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: stopping tablet replica
I20260928 14:51:37.451639 4398 raft_consensus.cc:2229] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:51:37.451678 4032 tablet_service.cc:1514] Processing DeleteTablet for tablet 552fc69d52c1458aaa336cf74ee7c75d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:37 UTC) from {username='slave'} at 127.0.0.1:46038
I20260928 14:51:37.451751 4398 raft_consensus.cc:2258] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:37.451841 4399 raft_consensus.cc:2229] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:51:37.452028 4399 raft_consensus.cc:2258] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:37.452073 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 3970
I20260928 14:51:37.453302 4399 ts_tablet_manager.cc:1905] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:37.455288 4399 ts_tablet_manager.cc:1918] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:37.455392 4399 log.cc:1196] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/552fc69d52c1458aaa336cf74ee7c75d
I20260928 14:51:37.455626 4399 ts_tablet_manager.cc:1939] T 552fc69d52c1458aaa336cf74ee7c75d P cb3e9f0cdb9545849155d4dc96f8aecf: Deleting consensus metadata
I20260928 14:51:37.456094 3894 catalog_manager.cc:4841] TS cb3e9f0cdb9545849155d4dc96f8aecf (127.18.96.130:34061): tablet 552fc69d52c1458aaa336cf74ee7c75d (table test-table [id=c705a7e058de40afb3f51bb04008cbe8]) successfully deleted
W20260928 14:51:37.457486 3894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:38993 (error 108)
I20260928 14:51:37.457830 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 4102
I20260928 14:51:37.457849 4398 ts_tablet_manager.cc:1905] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:37.459929 4398 ts_tablet_manager.cc:1918] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:37.460012 4398 log.cc:1196] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/552fc69d52c1458aaa336cf74ee7c75d
I20260928 14:51:37.460290 4398 ts_tablet_manager.cc:1939] T 552fc69d52c1458aaa336cf74ee7c75d P 25b78ab1dc1647ddabc26b45183c0bbb: Deleting consensus metadata
I20260928 14:51:37.460752 3897 catalog_manager.cc:4841] TS 25b78ab1dc1647ddabc26b45183c0bbb (127.18.96.131:39505): tablet 552fc69d52c1458aaa336cf74ee7c75d (table test-table [id=c705a7e058de40afb3f51bb04008cbe8]) successfully deleted
I20260928 14:51:37.463667 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 4234
I20260928 14:51:37.463915 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:37.463680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:38993 (local address 127.0.0.1:46068)
0928 14:51:37.463783 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:37.463847 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 14:51:37.465529 3894 catalog_manager.cc:4584] TS 086d55307b4c47c9a945ba96cc85420d (127.18.96.129:38993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 552fc69d52c1458aaa336cf74ee7c75d: Network error: Client connection negotiation failed: client connection to 127.18.96.129:38993: connect: Connection refused (error 111)
I20260928 14:51:37.468185 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 3878
2026-09-28T14:51:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (857 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:37 dist-test-slave-j968 krb5kdc[4404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:37 dist-test-slave-j968 krb5kdc[4404](info): set up 2 sockets
Sep 28 14:51:37 dist-test-slave-j968 krb5kdc[4404](info): commencing operation
krb5kdc: starting...
W20260928 14:51:38.537290 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:38 dist-test-slave-j968 krb5kdc[4404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:51:38.664973 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:44395
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:44395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:38.747773 4417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:38.748085 4417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:38.748127 4417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:38.749325 4417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:38.749392 4417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:38.749415 4417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:38.749435 4417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:38.749467 4417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:38.750989 4417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:44395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:44395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:38.751283 4417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:38.751510 4417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:38.756878 4422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:38.756886 4425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:38.756886 4423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:38.757220 4417 server_base.cc:1034] running on GCE node
I20260928 14:51:38.757503 4417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:38.757789 4417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:38.758946 4417 hybrid_clock.cc:648] HybridClock initialized: now 1790607098758894 us; error 74 us; skew 500 ppm
Sep 28 14:51:38 dist-test-slave-j968 krb5kdc[4404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607098, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:38.761552 4417 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:51:38.762060 4417 webserver.cc:460] Webserver started at http://127.18.96.190:40555/ using document root <none> and password file <none>
I20260928 14:51:38.762306 4417 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:38.762369 4417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:38.762506 4417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:38.763567 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "d999fb12acef46a6ae2b224355e4deca"
format_stamp: "Formatted at 2026-09-28 14:51:38 on dist-test-slave-j968"
server_key: "3e4bc570cb535395bf3ca6d8efcc4ca9"
server_key_iv: "abc5f835d7f523070b37cea4bc6ed9c6"
server_key_version: "encryptionkey@0"
I20260928 14:51:38.763993 4417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "d999fb12acef46a6ae2b224355e4deca"
format_stamp: "Formatted at 2026-09-28 14:51:38 on dist-test-slave-j968"
server_key: "3e4bc570cb535395bf3ca6d8efcc4ca9"
server_key_iv: "abc5f835d7f523070b37cea4bc6ed9c6"
server_key_version: "encryptionkey@0"
I20260928 14:51:38.765424 4417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:38.766147 4432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:38.766358 4417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:51:38.766448 4417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "d999fb12acef46a6ae2b224355e4deca"
format_stamp: "Formatted at 2026-09-28 14:51:38 on dist-test-slave-j968"
server_key: "3e4bc570cb535395bf3ca6d8efcc4ca9"
server_key_iv: "abc5f835d7f523070b37cea4bc6ed9c6"
server_key_version: "encryptionkey@0"
I20260928 14:51:38.766523 4417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:38.787685 4417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:38.789608 4417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:38.789798 4417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:38.793843 4417 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:44395
I20260928 14:51:38.793886 4484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:44395 every 8 connection(s)
I20260928 14:51:38.794255 4417 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:38.795101 4485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:38.797478 4485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca: Bootstrap starting.
I20260928 14:51:38.798015 4485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:38.798256 4485 log.cc:824] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:38.798892 4485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca: No bootstrap required, opened a new log
I20260928 14:51:38.800369 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 4417
I20260928 14:51:38.800477 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:38.800676 18818 external_mini_cluster.cc:1428] Setting key 1461ef5ae17979bf95168cf2c5e66683
I20260928 14:51:38.800899 4485 raft_consensus.cc:348] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d999fb12acef46a6ae2b224355e4deca" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 44395 } }
I20260928 14:51:38.801056 4485 raft_consensus.cc:374] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:38.801091 4485 raft_consensus.cc:729] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d999fb12acef46a6ae2b224355e4deca, State: Initialized, Role: FOLLOWER
I20260928 14:51:38.801450 4485 consensus_queue.cc:260] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d999fb12acef46a6ae2b224355e4deca" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 44395 } }
I20260928 14:51:38.801546 4485 raft_consensus.cc:388] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:38.801577 4485 raft_consensus.cc:482] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:38.801625 4485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:38.802616 4485 raft_consensus.cc:504] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d999fb12acef46a6ae2b224355e4deca" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 44395 } }
I20260928 14:51:38.802721 4485 leader_election.cc:302] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d999fb12acef46a6ae2b224355e4deca; no voters:
I20260928 14:51:38.802874 4485 leader_election.cc:288] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:38.802927 4490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:38.803071 4490 raft_consensus.cc:686] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [term 1 LEADER]: Becoming Leader. State: Replica: d999fb12acef46a6ae2b224355e4deca, State: Running, Role: LEADER
I20260928 14:51:38.803121 4485 sys_catalog.cc:564] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:38.803169 4490 consensus_queue.cc:237] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d999fb12acef46a6ae2b224355e4deca" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 44395 } }
I20260928 14:51:38.803597 4491 sys_catalog.cc:455] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d999fb12acef46a6ae2b224355e4deca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d999fb12acef46a6ae2b224355e4deca" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 44395 } } }
I20260928 14:51:38.803692 4491 sys_catalog.cc:458] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:38.803901 4499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:38.803936 4492 sys_catalog.cc:455] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [sys.catalog]: SysCatalogTable state changed. Reason: New leader d999fb12acef46a6ae2b224355e4deca. Latest consensus state: current_term: 1 leader_uuid: "d999fb12acef46a6ae2b224355e4deca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d999fb12acef46a6ae2b224355e4deca" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 44395 } } }
I20260928 14:51:38.804093 4492 sys_catalog.cc:458] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:38.804437 4499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:38.806187 4499 catalog_manager.cc:1347] Generated new cluster ID: 8b7c9015b86c4030812edb9aceac440c
I20260928 14:51:38.806243 4499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:51:38 dist-test-slave-j968 krb5kdc[4404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:38.812750 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:38.801222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.18.96.190:44395)
0928 14:51:38.801387 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:38.801397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:38.801432 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:51:38.801591 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:38.801599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:38.801657 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:38.801821 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:38.802460 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:38.803913 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:38.805099 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:38.805623 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:38.805747 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:38.808944 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:38.808949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:38.808949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:38.808963 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:38.810940 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:38.811327 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:38.811332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:38.811332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:38.811487 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:38.811698 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:38.811699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:38.811699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:38.812091 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:38.812304 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:38.812384 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 14:51:38.812530 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260928 14:51:38.830237 4499 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:38.831005 4499 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:38.838826 4499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca: Generated new TSK 0
I20260928 14:51:38.839051 4499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:51:38.870663 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:44395
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:38.954344 4511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:38.954577 4511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:38.954619 4511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:38.955912 4511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:38.955972 4511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:38.956018 4511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:51:38.957547 4511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:44395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:38.957821 4511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:38.958055 4511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:38.963708 4517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:38.963719 4519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:38.963719 4516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:38.964103 4511 server_base.cc:1034] running on GCE node
I20260928 14:51:38.964337 4511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:38.964597 4511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:38.965755 4511 hybrid_clock.cc:648] HybridClock initialized: now 1790607098965740 us; error 32 us; skew 500 ppm
Sep 28 14:51:38 dist-test-slave-j968 krb5kdc[4404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607098, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:38.968489 4511 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:51:38.969043 4511 webserver.cc:460] Webserver started at http://127.18.96.129:37115/ using document root <none> and password file <none>
I20260928 14:51:38.969297 4511 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:38.969352 4511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:38.969465 4511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:38.970639 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "6032aa5ebcd94e90b942d091253ec390"
format_stamp: "Formatted at 2026-09-28 14:51:38 on dist-test-slave-j968"
server_key: "6e027677de572bf9a17a826bbed10446"
server_key_iv: "3dff69f1a8e89e783d0c81b9cabd1cc8"
server_key_version: "encryptionkey@0"
I20260928 14:51:38.970947 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "6032aa5ebcd94e90b942d091253ec390"
format_stamp: "Formatted at 2026-09-28 14:51:38 on dist-test-slave-j968"
server_key: "6e027677de572bf9a17a826bbed10446"
server_key_iv: "3dff69f1a8e89e783d0c81b9cabd1cc8"
server_key_version: "encryptionkey@0"
I20260928 14:51:38.972337 4511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:51:38.973033 4526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:38.973265 4511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:51:38.973366 4511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "6032aa5ebcd94e90b942d091253ec390"
format_stamp: "Formatted at 2026-09-28 14:51:38 on dist-test-slave-j968"
server_key: "6e027677de572bf9a17a826bbed10446"
server_key_iv: "3dff69f1a8e89e783d0c81b9cabd1cc8"
server_key_version: "encryptionkey@0"
I20260928 14:51:38.973440 4511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:38.981667 4511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:38.983541 4511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:38.983698 4511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:38.983958 4511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:38.984264 4511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:38.984299 4511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:38.984320 4511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:38.984333 4511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:38.989993 4511 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:34927
I20260928 14:51:38.990072 4640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:34927 every 8 connection(s)
I20260928 14:51:38.990360 4511 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:51:38.996075 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 4511
I20260928 14:51:38.996204 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:51:38.996413 18818 external_mini_cluster.cc:1428] Setting key 44285c5df47d01d38b50a84194fb2e6c
Sep 28 14:51:38 dist-test-slave-j968 krb5kdc[4404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607098, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:39.000521 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:38.991079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:59163 (local address 127.18.96.190:44395)
0928 14:51:38.991142 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:38.991144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:38.991324 (+ 180us) server_negotiation.cc:407] Connection header received
0928 14:51:38.991815 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:38.991816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:38.991828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:38.991928 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:38.992882 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:38.993423 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:38.994503 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:38.994711 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:38.994767 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:38.997416 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:38.997418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:38.997418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:38.997426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:38.998851 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:38.999351 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:38.999352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:38.999353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:38.999414 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:38.999685 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:38.999686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:38.999686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:38.999904 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:38.999975 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.000280 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.000378 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:51:39.000604 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:38.991055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:44395 (local address 127.18.96.129:59163)
0928 14:51:38.991217 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:38.991254 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:38.991728 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:38.991998 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:38.992002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:38.992585 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:38.992787 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:38.992792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:38.993510 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:38.993511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:38.994424 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:38.994427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:38.994786 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:38.994787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:38.994892 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:38.995466 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:38.995474 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:38.997324 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:38.998932 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:38.998940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:38.998943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:38.999270 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:38.999488 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:38.999488 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:38.999489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:38.999621 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:38.999978 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:38.999986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:39.000119 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.000228 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.000374 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260928 14:51:39.001282 4641 heartbeater.cc:347] Connected to a master server at 127.18.96.190:44395
I20260928 14:51:39.001405 4641 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:39.001601 4641 heartbeater.cc:511] Master 127.18.96.190:44395 requested a full tablet report, sending...
I20260928 14:51:39.002187 4449 ts_manager.cc:194] Registered new tserver with Master: 6032aa5ebcd94e90b942d091253ec390 (127.18.96.129:34927)
I20260928 14:51:39.002864 4449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:59163
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:51:39.023805 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:44395
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:39.107702 4646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:39.107954 4646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:39.107982 4646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:39.109342 4646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:39.109395 4646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:39.109426 4646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:51:39.110950 4646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:44395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:39.111210 4646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:39.111469 4646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:39.116735 4651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:39.116786 4654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:39.116735 4652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:39.116873 4646 server_base.cc:1034] running on GCE node
I20260928 14:51:39.117257 4646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:39.117462 4646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:39.118590 4646 hybrid_clock.cc:648] HybridClock initialized: now 1790607099118582 us; error 32 us; skew 500 ppm
Sep 28 14:51:39 dist-test-slave-j968 krb5kdc[4404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607099, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:39.121269 4646 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:51:39.121713 4646 webserver.cc:460] Webserver started at http://127.18.96.130:41691/ using document root <none> and password file <none>
I20260928 14:51:39.121945 4646 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:39.122002 4646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:39.122228 4646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:39.123395 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "b206c7948cee492d95b6cafb4d88943d"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "81b43779e252437b8ed0dc84d5586172"
server_key_iv: "60c7bffa7fdc9d9d8582d523b1490833"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.123805 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "b206c7948cee492d95b6cafb4d88943d"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "81b43779e252437b8ed0dc84d5586172"
server_key_iv: "60c7bffa7fdc9d9d8582d523b1490833"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.125296 4646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:39.126087 4661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.126294 4646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:51:39.126389 4646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "b206c7948cee492d95b6cafb4d88943d"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "81b43779e252437b8ed0dc84d5586172"
server_key_iv: "60c7bffa7fdc9d9d8582d523b1490833"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.126457 4646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:39.133895 4646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:39.136391 4646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:39.136554 4646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:39.136790 4646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:39.137107 4646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:39.137141 4646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.137182 4646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:39.137208 4646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.143155 4646 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:34397
I20260928 14:51:39.143239 4774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:34397 every 8 connection(s)
I20260928 14:51:39.143558 4646 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:51:39.148859 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 4646
I20260928 14:51:39.148957 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:51:39.149209 18818 external_mini_cluster.cc:1428] Setting key ab9e1d53c8786951a4faf6aeff724b58
Sep 28 14:51:39 dist-test-slave-j968 krb5kdc[4404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607099, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:39.153381 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.144336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:43997 (local address 127.18.96.190:44395)
0928 14:51:39.144399 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.144401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.144595 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:51:39.145084 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.145085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.145096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.145157 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:39.146102 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.146612 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.147708 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.147959 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.148025 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.150348 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:39.150350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:39.150350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:39.150357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:39.151974 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:39.152406 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:39.152407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:39.152408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:39.152471 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:39.152728 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:39.152729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:39.152729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:39.152848 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:39.152913 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.153195 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.153290 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:51:39.153594 4777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.144305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:44395 (local address 127.18.96.130:43997)
0928 14:51:39.144461 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.144506 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.144999 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.145211 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:39.145215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:39.145781 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:39.145998 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.146002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.146701 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.146702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.147630 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.147633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.148034 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.148035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.148188 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.148650 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:39.148659 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:39.150254 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:39.152075 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:39.152083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:39.152085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:39.152316 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:39.152535 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:39.152536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:39.152536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:39.152657 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:39.152916 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:39.152923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:39.153068 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.153174 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.153337 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260928 14:51:39.154158 4775 heartbeater.cc:347] Connected to a master server at 127.18.96.190:44395
I20260928 14:51:39.154243 4775 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:39.154439 4775 heartbeater.cc:511] Master 127.18.96.190:44395 requested a full tablet report, sending...
I20260928 14:51:39.154808 4449 ts_manager.cc:194] Registered new tserver with Master: b206c7948cee492d95b6cafb4d88943d (127.18.96.130:34397)
I20260928 14:51:39.155189 4449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:43997
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:51:39.179626 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:44395
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:39.264679 4780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:39.264925 4780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:39.264962 4780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:39.266140 4780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:39.266211 4780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:39.266256 4780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:51:39.267709 4780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:44395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:39.267992 4780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:39.268227 4780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:39.273593 4788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:39.273681 4785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:39.273593 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:39.273856 4780 server_base.cc:1034] running on GCE node
I20260928 14:51:39.274045 4780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:39.274253 4780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:39.275456 4780 hybrid_clock.cc:648] HybridClock initialized: now 1790607099275421 us; error 28 us; skew 500 ppm
Sep 28 14:51:39 dist-test-slave-j968 krb5kdc[4404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607099, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:39.278126 4780 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:51:39.278592 4780 webserver.cc:460] Webserver started at http://127.18.96.131:45317/ using document root <none> and password file <none>
I20260928 14:51:39.278810 4780 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:39.278873 4780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:39.279016 4780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:39.280176 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "23beda68735306e426d4773aed83f84b"
server_key_iv: "83e3e0d8969f09ffbed12946eb162d56"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.280524 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "23beda68735306e426d4773aed83f84b"
server_key_iv: "83e3e0d8969f09ffbed12946eb162d56"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.281852 4780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:39.282622 4795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.282784 4780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:51:39.282882 4780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "23beda68735306e426d4773aed83f84b"
server_key_iv: "83e3e0d8969f09ffbed12946eb162d56"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.282956 4780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:39.297518 4780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:39.299835 4780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:39.300093 4780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:39.300329 4780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:39.300662 4780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:39.300714 4780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.300750 4780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:39.300777 4780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.307396 4780 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:36527
I20260928 14:51:39.307437 4908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:36527 every 8 connection(s)
I20260928 14:51:39.307776 4780 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:51:39 dist-test-slave-j968 krb5kdc[4404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607099, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:39.315166 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 4780
I20260928 14:51:39.315295 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:51:39.315526 18818 external_mini_cluster.cc:1428] Setting key 0994f04259792cce0cfe5d10c7a9d261
I20260928 14:51:39.317859 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.308553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:41513 (local address 127.18.96.190:44395)
0928 14:51:39.308598 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.308600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.308816 (+ 216us) server_negotiation.cc:407] Connection header received
0928 14:51:39.309415 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.309416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.309430 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.309498 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:39.310482 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.310989 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.312045 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.312249 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.312323 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.314704 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:39.314706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:39.314706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:39.314714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:39.316442 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:39.316912 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:39.316913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:39.316913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:39.316968 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:39.317260 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:39.317261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:39.317261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:39.317387 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:39.317453 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.317724 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.317795 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 14:51:39.318085 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.308535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:44395 (local address 127.18.96.131:41513)
0928 14:51:39.308690 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.308728 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.309305 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.309571 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:39.309575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:39.310183 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:39.310384 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.310388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.311084 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.311085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.311952 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.311954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.312329 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.312331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.312477 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.313090 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:39.313101 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:39.314604 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:39.316519 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:39.316526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:39.316529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:39.316845 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:39.317045 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:39.317046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:39.317047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:39.317174 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:39.317453 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:39.317458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:39.317603 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.317692 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.317822 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260928 14:51:39.318557 4909 heartbeater.cc:347] Connected to a master server at 127.18.96.190:44395
I20260928 14:51:39.318656 4909 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:39.318835 4909 heartbeater.cc:511] Master 127.18.96.190:44395 requested a full tablet report, sending...
I20260928 14:51:39.319203 4449 ts_manager.cc:194] Registered new tserver with Master: 45bb8ec551fd42d3a9c4ce2ca9918ef7 (127.18.96.131:36527)
I20260928 14:51:39.319581 4449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:41513
I20260928 14:51:39.327481 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:51:39.335961 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.329029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.18.96.190:44395)
0928 14:51:39.329104 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.329106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.329257 (+ 151us) server_negotiation.cc:407] Connection header received
0928 14:51:39.329366 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.329367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.329378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.329433 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:39.330028 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.330597 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.331348 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.331555 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.331635 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.333074 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:39.333076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:39.333076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:39.333085 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:39.334600 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:39.335067 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:39.335068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:39.335069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:39.335134 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:39.335351 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:39.335353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:39.335353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:39.335519 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:39.335638 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.335753 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.335817 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:51:39.338012 4449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:51:39.340054 4449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:51:39.347020 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.343032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.18.96.129:34927)
0928 14:51:39.343202 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.343206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.343217 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:51:39.343273 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.343275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.343291 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.343611 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:39.343691 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.344470 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.346005 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.346594 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.346681 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.346744 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.346886 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.346926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260928 14:51:39.347009 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.342809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:34927 (local address 127.0.0.1:33134)
0928 14:51:39.343030 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.343039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.343081 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.343603 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:39.343604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:39.343605 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:39.343639 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.343640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.344587 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.344588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.345901 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.345903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.346680 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.346681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.346759 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.346773 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.346867 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.346925 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":60,"threads_started":1}
I20260928 14:51:39.347313 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.342687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:34397 (local address 127.0.0.1:33772)
0928 14:51:39.342910 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.342923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.342977 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.343392 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:39.343395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:39.343398 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:39.343491 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.343494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.344989 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.344990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.346054 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.346056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.346938 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.346939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.347005 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.347010 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.347040 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.347228 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260928 14:51:39.347424 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.342696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.18.96.131:36527)
0928 14:51:39.343527 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.343547 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.343556 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:51:39.343632 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.343635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.343650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.343733 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:39.343809 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.344901 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.346072 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.346833 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.346907 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.346961 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.347276 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.347330 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":84,"threads_started":1}
I20260928 14:51:39.347729 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.342860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.18.96.130:34397)
0928 14:51:39.343207 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.343211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.343220 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:51:39.343269 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.343271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.343283 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.343382 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:39.343650 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.344878 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.346135 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.346833 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.347494 (+ 661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.347569 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.347599 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.347645 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":79,"threads_started":1}
I20260928 14:51:39.347816 4575 tablet_service.cc:1467] Processing CreateTablet for tablet 89a6641ba6ed48d9afa194726b6b878f (DEFAULT_TABLE table=test-table [id=dc048197c9f1445f9350a70d94702aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:39.348083 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.342611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:36527 (local address 127.0.0.1:58494)
0928 14:51:39.342826 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.342875 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.342949 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.343725 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:39.343726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:39.343728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:39.343759 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.343760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.345010 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.345011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.345963 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.345966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.346989 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.346990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.347047 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.347049 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.347107 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.347233 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":5,"thread_start_us":54,"threads_started":1}
I20260928 14:51:39.348265 4575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a6641ba6ed48d9afa194726b6b878f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:39.348383 4709 tablet_service.cc:1467] Processing CreateTablet for tablet 89a6641ba6ed48d9afa194726b6b878f (DEFAULT_TABLE table=test-table [id=dc048197c9f1445f9350a70d94702aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:39.348695 4709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a6641ba6ed48d9afa194726b6b878f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:39.350878 4926 tablet_bootstrap.cc:492] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390: Bootstrap starting.
I20260928 14:51:39.351248 4927 tablet_bootstrap.cc:492] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d: Bootstrap starting.
I20260928 14:51:39.351675 4926 tablet_bootstrap.cc:654] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:39.351760 4843 tablet_service.cc:1467] Processing CreateTablet for tablet 89a6641ba6ed48d9afa194726b6b878f (DEFAULT_TABLE table=test-table [id=dc048197c9f1445f9350a70d94702aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:39.351933 4927 tablet_bootstrap.cc:654] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:39.352178 4843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a6641ba6ed48d9afa194726b6b878f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:39.352308 4926 log.cc:824] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:39.352490 4927 log.cc:824] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:39.353153 4926 tablet_bootstrap.cc:492] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390: No bootstrap required, opened a new log
I20260928 14:51:39.353223 4926 ts_tablet_manager.cc:1397] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:51:39.353775 4927 tablet_bootstrap.cc:492] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d: No bootstrap required, opened a new log
I20260928 14:51:39.353909 4927 ts_tablet_manager.cc:1397] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:51:39.355376 4927 raft_consensus.cc:348] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } }
I20260928 14:51:39.355298 4926 raft_consensus.cc:348] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } }
I20260928 14:51:39.355541 4927 raft_consensus.cc:374] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:39.355541 4926 raft_consensus.cc:374] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:39.355583 4926 raft_consensus.cc:729] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6032aa5ebcd94e90b942d091253ec390, State: Initialized, Role: FOLLOWER
I20260928 14:51:39.355582 4927 raft_consensus.cc:729] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b206c7948cee492d95b6cafb4d88943d, State: Initialized, Role: FOLLOWER
I20260928 14:51:39.355708 4927 consensus_queue.cc:260] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } }
I20260928 14:51:39.355705 4926 consensus_queue.cc:260] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } }
I20260928 14:51:39.356024 4926 ts_tablet_manager.cc:1428] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 14:51:39.356041 4775 heartbeater.cc:503] Master 127.18.96.190:44395 was elected leader, sending a full tablet report...
I20260928 14:51:39.356024 4930 tablet_bootstrap.cc:492] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: Bootstrap starting.
I20260928 14:51:39.356024 4927 ts_tablet_manager.cc:1428] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:51:39.356149 4641 heartbeater.cc:503] Master 127.18.96.190:44395 was elected leader, sending a full tablet report...
I20260928 14:51:39.356850 4930 tablet_bootstrap.cc:654] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:39.357486 4930 log.cc:824] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:39.358171 4930 tablet_bootstrap.cc:492] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: No bootstrap required, opened a new log
I20260928 14:51:39.358253 4930 ts_tablet_manager.cc:1397] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:51:39.359634 4930 raft_consensus.cc:348] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } }
I20260928 14:51:39.359795 4930 raft_consensus.cc:374] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:39.359843 4930 raft_consensus.cc:729] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bb8ec551fd42d3a9c4ce2ca9918ef7, State: Initialized, Role: FOLLOWER
I20260928 14:51:39.359987 4930 consensus_queue.cc:260] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } }
I20260928 14:51:39.360246 4930 ts_tablet_manager.cc:1428] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:51:39.360303 4909 heartbeater.cc:503] Master 127.18.96.190:44395 was elected leader, sending a full tablet report...
W20260928 14:51:39.394255 4776 tablet.cc:2369] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:51:39.418709 4934 raft_consensus.cc:482] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:39.418883 4934 raft_consensus.cc:504] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } }
I20260928 14:51:39.419301 4934 leader_election.cc:288] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6032aa5ebcd94e90b942d091253ec390 (127.18.96.129:34927), b206c7948cee492d95b6cafb4d88943d (127.18.96.130:34397)
I20260928 14:51:39.423310 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.419673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:42579 (local address 127.18.96.130:34397)
0928 14:51:39.419730 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.419732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.419784 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:51:39.419897 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.419898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.419915 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.419984 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:39.420206 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.420889 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.422323 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.422920 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.423003 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.423012 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.423181 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.423238 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:51:39.423319 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.419468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:34927 (local address 127.18.96.131:45281)
0928 14:51:39.419535 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.419565 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.419619 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.419788 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:39.419789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:39.419791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:39.419892 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.419894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.420843 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.420844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.422209 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.422211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.423010 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.423011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.423176 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.423179 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.423228 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.423261 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":21}
I20260928 14:51:39.423370 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.419602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:45281 (local address 127.18.96.129:34927)
0928 14:51:39.419653 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.419655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.419666 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:51:39.419698 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.419699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.419713 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.419781 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:39.419967 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.420748 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.422323 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.422920 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.423003 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.423012 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.423251 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.423294 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:51:39.423322 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.419517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:34397 (local address 127.18.96.131:42579)
0928 14:51:39.419723 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.419733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.419814 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.420033 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:39.420034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:39.420036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:39.420133 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.420135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.420972 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.420973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.422208 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.422211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.423010 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.423011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.423102 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.423106 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.423169 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.423219 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":37,"thread_start_us":114,"threads_started":1}
I20260928 14:51:39.423840 4595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a6641ba6ed48d9afa194726b6b878f" candidate_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6032aa5ebcd94e90b942d091253ec390" is_pre_election: true
I20260928 14:51:39.424064 4595 raft_consensus.cc:2454] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45bb8ec551fd42d3a9c4ce2ca9918ef7 in term 0.
I20260928 14:51:39.424126 4729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a6641ba6ed48d9afa194726b6b878f" candidate_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b206c7948cee492d95b6cafb4d88943d" is_pre_election: true
I20260928 14:51:39.424279 4729 raft_consensus.cc:2454] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45bb8ec551fd42d3a9c4ce2ca9918ef7 in term 0.
I20260928 14:51:39.424312 4799 leader_election.cc:302] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45bb8ec551fd42d3a9c4ce2ca9918ef7, 6032aa5ebcd94e90b942d091253ec390; no voters:
I20260928 14:51:39.424443 4934 raft_consensus.cc:2781] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:51:39.424520 4934 raft_consensus.cc:482] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:51:39.424551 4934 raft_consensus.cc:3037] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:39.425269 4934 raft_consensus.cc:504] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } }
I20260928 14:51:39.425419 4934 leader_election.cc:288] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 6032aa5ebcd94e90b942d091253ec390 (127.18.96.129:34927), b206c7948cee492d95b6cafb4d88943d (127.18.96.130:34397)
I20260928 14:51:39.425570 4595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a6641ba6ed48d9afa194726b6b878f" candidate_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6032aa5ebcd94e90b942d091253ec390"
I20260928 14:51:39.425576 4729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a6641ba6ed48d9afa194726b6b878f" candidate_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b206c7948cee492d95b6cafb4d88943d"
I20260928 14:51:39.425668 4595 raft_consensus.cc:3037] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:39.425670 4729 raft_consensus.cc:3037] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:39.426400 4595 raft_consensus.cc:2454] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bb8ec551fd42d3a9c4ce2ca9918ef7 in term 1.
I20260928 14:51:39.426558 4729 raft_consensus.cc:2454] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bb8ec551fd42d3a9c4ce2ca9918ef7 in term 1.
I20260928 14:51:39.426644 4799 leader_election.cc:302] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45bb8ec551fd42d3a9c4ce2ca9918ef7, 6032aa5ebcd94e90b942d091253ec390; no voters:
I20260928 14:51:39.426744 4934 raft_consensus.cc:2781] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:39.426900 4934 raft_consensus.cc:686] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 45bb8ec551fd42d3a9c4ce2ca9918ef7, State: Running, Role: LEADER
I20260928 14:51:39.427012 4934 consensus_queue.cc:237] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } }
I20260928 14:51:39.427973 4449 catalog_manager.cc:5495] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45bb8ec551fd42d3a9c4ce2ca9918ef7 (127.18.96.131). New cstate: current_term: 1 leader_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45bb8ec551fd42d3a9c4ce2ca9918ef7" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 36527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 } health_report { overall_health: UNKNOWN } } }
I20260928 14:51:39.444218 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.441423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.18.96.131:36527)
0928 14:51:39.441483 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.441485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.441492 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:51:39.441591 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.441592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.441611 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.441679 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:51:39.441837 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.442390 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.443242 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.443436 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.443501 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.443770 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:51:39.443921 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:51:39.444018 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.444063 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.444076 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:51:39.446309 4595 raft_consensus.cc:1261] T 89a6641ba6ed48d9afa194726b6b878f P 6032aa5ebcd94e90b942d091253ec390 [term 1 FOLLOWER]: Refusing update from remote peer 45bb8ec551fd42d3a9c4ce2ca9918ef7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:39.446311 4729 raft_consensus.cc:1261] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [term 1 FOLLOWER]: Refusing update from remote peer 45bb8ec551fd42d3a9c4ce2ca9918ef7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:39.446681 4934 consensus_queue.cc:1035] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6032aa5ebcd94e90b942d091253ec390" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 34927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:39.446915 4936 consensus_queue.cc:1035] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b206c7948cee492d95b6cafb4d88943d" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 34397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:39.449292 4939 mvcc.cc:205] Tried to move back new op lower bound from 7334326679329120256 to 7334326679253868544. Current Snapshot: MvccSnapshot[applied={T|T < 7334326679329120256}]
I20260928 14:51:39.450178 4941 mvcc.cc:205] Tried to move back new op lower bound from 7334326679329120256 to 7334326679253868544. Current Snapshot: MvccSnapshot[applied={T|T < 7334326679329120256 or (T in {7334326679329120256})}]
I20260928 14:51:39.451675 4449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:39.451773 4449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:39.452569 4449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d999fb12acef46a6ae2b224355e4deca: Sending DeleteTablet for 3 replicas of tablet 89a6641ba6ed48d9afa194726b6b878f
I20260928 14:51:39.452911 4843 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a6641ba6ed48d9afa194726b6b878f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:39 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:58494
I20260928 14:51:39.452970 4709 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a6641ba6ed48d9afa194726b6b878f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:39 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:33772
I20260928 14:51:39.453110 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 4511
I20260928 14:51:39.453118 4575 tablet_service.cc:1514] Processing DeleteTablet for tablet 89a6641ba6ed48d9afa194726b6b878f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:39 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:33134
I20260928 14:51:39.454180 4949 tablet_replica.cc:330] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: stopping tablet replica
I20260928 14:51:39.454283 4949 raft_consensus.cc:2229] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:51:39.454371 4949 raft_consensus.cc:2258] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:39.454802 4949 ts_tablet_manager.cc:1905] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:39.456506 4949 ts_tablet_manager.cc:1918] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:39.456578 4949 log.cc:1196] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/89a6641ba6ed48d9afa194726b6b878f
I20260928 14:51:39.456741 4950 tablet_replica.cc:330] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d: stopping tablet replica
I20260928 14:51:39.456851 4950 raft_consensus.cc:2229] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:51:39.456863 4949 ts_tablet_manager.cc:1939] T 89a6641ba6ed48d9afa194726b6b878f P 45bb8ec551fd42d3a9c4ce2ca9918ef7: Deleting consensus metadata
I20260928 14:51:39.456965 4950 raft_consensus.cc:2258] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:39.457259 4436 catalog_manager.cc:4841] TS 45bb8ec551fd42d3a9c4ce2ca9918ef7 (127.18.96.131:36527): tablet 89a6641ba6ed48d9afa194726b6b878f (table test-table [id=dc048197c9f1445f9350a70d94702aae]) successfully deleted
W20260928 14:51:39.459149 4436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:34927 (error 108)
W20260928 14:51:39.460420 4920 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:51:39.459545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:34927 (local address 127.0.0.1:33136)
0928 14:51:39.459607 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.459618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.459659 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.460363 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:34927: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:51:39.460582 4436 catalog_manager.cc:4584] TS 6032aa5ebcd94e90b942d091253ec390 (127.18.96.129:34927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89a6641ba6ed48d9afa194726b6b878f: Network error: Client connection negotiation failed: client connection to 127.18.96.129:34927: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:51:39.460821 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 4646
I20260928 14:51:39.461015 4950 ts_tablet_manager.cc:1905] T 89a6641ba6ed48d9afa194726b6b878f P b206c7948cee492d95b6cafb4d88943d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:39.466737 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 4780
I20260928 14:51:39.467489 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.466793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:34397 (local address 127.0.0.1:33786)
0928 14:51:39.467369 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.467427 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.130:34397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552}
I20260928 14:51:39.472010 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 4417
2026-09-28T14:51:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2004 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T14:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:39Z Disabled control of system clock
I20260928 14:51:39.500252 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:40223
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:43367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:40223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:51:39.583757 4955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:39.584089 4955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:39.584117 4955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:39.585263 4955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:39.585310 4955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:39.585323 4955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:39.585336 4955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:39.585348 4955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:39.586679 4955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:40223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:40223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:39.586890 4955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:39.587155 4955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:39.592057 4960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:39.592067 4961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:39.592185 4963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:39.592489 4955 server_base.cc:1034] running on GCE node
I20260928 14:51:39.592703 4955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:39.592957 4955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:39.594101 4955 hybrid_clock.cc:648] HybridClock initialized: now 1790607099594086 us; error 29 us; skew 500 ppm
I20260928 14:51:39.595369 4955 webserver.cc:460] Webserver started at http://127.18.96.190:41309/ using document root <none> and password file <none>
I20260928 14:51:39.595639 4955 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:39.595685 4955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:39.595901 4955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:39.597038 4955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "d59e333869b84e029bf2b45e49f279b1"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "0ca8b3bc79ad27a7251bbc2c3428db36"
server_key_iv: "492b9aa09dcb2a74cb03953a78c3246e"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.597441 4955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "d59e333869b84e029bf2b45e49f279b1"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "0ca8b3bc79ad27a7251bbc2c3428db36"
server_key_iv: "492b9aa09dcb2a74cb03953a78c3246e"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.598850 4955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:51:39.599702 4969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.599915 4955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:39.600005 4955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "d59e333869b84e029bf2b45e49f279b1"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "0ca8b3bc79ad27a7251bbc2c3428db36"
server_key_iv: "492b9aa09dcb2a74cb03953a78c3246e"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.600082 4955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:39.634783 4955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:39.635092 4955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:39.635203 4955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:39.639178 4955 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:40223
I20260928 14:51:39.639237 5021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:40223 every 8 connection(s)
I20260928 14:51:39.639598 4955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:39.640727 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:39.643092 5022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1: Bootstrap starting.
I20260928 14:51:39.643637 5022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:39.643900 5022 log.cc:824] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:39.644583 5022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1: No bootstrap required, opened a new log
I20260928 14:51:39.644991 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 4955
I20260928 14:51:39.645150 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:39.645324 18818 external_mini_cluster.cc:1428] Setting key 2682999653870d8d0f3196061e02f11c
I20260928 14:51:39.646713 5022 raft_consensus.cc:348] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59e333869b84e029bf2b45e49f279b1" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40223 } }
I20260928 14:51:39.646884 5022 raft_consensus.cc:374] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:39.646922 5022 raft_consensus.cc:729] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d59e333869b84e029bf2b45e49f279b1, State: Initialized, Role: FOLLOWER
I20260928 14:51:39.647105 5022 consensus_queue.cc:260] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59e333869b84e029bf2b45e49f279b1" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40223 } }
I20260928 14:51:39.647194 5022 raft_consensus.cc:388] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:39.647219 5022 raft_consensus.cc:482] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:39.647285 5022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:39.647830 5022 raft_consensus.cc:504] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59e333869b84e029bf2b45e49f279b1" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40223 } }
I20260928 14:51:39.647975 5022 leader_election.cc:302] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d59e333869b84e029bf2b45e49f279b1; no voters:
I20260928 14:51:39.648180 5022 leader_election.cc:288] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:39.648213 5027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:39.648320 5027 raft_consensus.cc:686] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [term 1 LEADER]: Becoming Leader. State: Replica: d59e333869b84e029bf2b45e49f279b1, State: Running, Role: LEADER
I20260928 14:51:39.648450 5027 consensus_queue.cc:237] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59e333869b84e029bf2b45e49f279b1" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40223 } }
I20260928 14:51:39.648844 5027 sys_catalog.cc:455] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d59e333869b84e029bf2b45e49f279b1. Latest consensus state: current_term: 1 leader_uuid: "d59e333869b84e029bf2b45e49f279b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59e333869b84e029bf2b45e49f279b1" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40223 } } }
I20260928 14:51:39.648872 5028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d59e333869b84e029bf2b45e49f279b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59e333869b84e029bf2b45e49f279b1" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 40223 } } }
I20260928 14:51:39.648450 5022 sys_catalog.cc:564] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:39.649060 5028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:39.649008 5027 sys_catalog.cc:458] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1 [sys.catalog]: This master's current role is: LEADER
W20260928 14:51:39.650828 5043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:51:39.650916 5043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:51:39.651010 5031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:39.651180 5031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:39.651996 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.645853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.18.96.190:40223)
0928 14:51:39.646077 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.646085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.646110 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:51:39.646689 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.646696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.646757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.646875 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:39.647171 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.648803 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.649853 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.650273 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.650404 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.651127 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:39.651130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:39.651131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:39.651335 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:39.651378 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:39.651393 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:39.651459 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.651667 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.651751 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260928 14:51:39.653028 5031 catalog_manager.cc:1347] Generated new cluster ID: ab2fdbcfba334c4c96fab1da530a50bb
I20260928 14:51:39.653082 5031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:39.667562 5031 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:39.668222 5031 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:39.688657 5031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1: Generated new TSK 0
I20260928 14:51:39.688890 5031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:39.692809 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40223
--builtin_ntp_servers=127.18.96.148:43367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:51:39.791625 5046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:39.791867 5046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:39.791923 5046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:39.793207 5046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:39.793277 5046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:39.793319 5046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:51:39.794749 5046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:39.795027 5046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:39.795336 5046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:39.800684 5051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:39.800664 5052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:39.800695 5046 server_base.cc:1034] running on GCE node
W20260928 14:51:39.800664 5054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:39.801115 5046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:39.801345 5046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:39.802498 5046 hybrid_clock.cc:648] HybridClock initialized: now 1790607099802470 us; error 28 us; skew 500 ppm
I20260928 14:51:39.804044 5046 webserver.cc:460] Webserver started at http://127.18.96.129:41273/ using document root <none> and password file <none>
I20260928 14:51:39.804289 5046 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:39.804337 5046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:39.804461 5046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:39.805577 5046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "7bd8df4433ad4c73bcb5b0991e938544"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "641e5be991d4f4646d953225a60efcf3"
server_key_iv: "a2db17ba3941806c2d3669439471c6e1"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.805933 5046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "7bd8df4433ad4c73bcb5b0991e938544"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "641e5be991d4f4646d953225a60efcf3"
server_key_iv: "a2db17ba3941806c2d3669439471c6e1"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.807395 5046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:51:39.808313 5060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.808480 5046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:51:39.808604 5046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "7bd8df4433ad4c73bcb5b0991e938544"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "641e5be991d4f4646d953225a60efcf3"
server_key_iv: "a2db17ba3941806c2d3669439471c6e1"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.808672 5046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:39.820099 5046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:39.820411 5046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:39.820549 5046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:39.820783 5046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:39.821117 5046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:39.821167 5046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.821209 5046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:39.821238 5046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.827095 5046 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:37235
I20260928 14:51:39.827222 5173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:37235 every 8 connection(s)
I20260928 14:51:39.827550 5046 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:51:39.828649 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 5046
I20260928 14:51:39.828778 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:51:39.828970 18818 external_mini_cluster.cc:1428] Setting key 4e3471c3bbfede4e47bf180f8c24d6d9
I20260928 14:51:39.829876 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40223
--builtin_ntp_servers=127.18.96.148:43367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:51:39.833467 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.828455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:50007 (local address 127.18.96.190:40223)
0928 14:51:39.828541 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.828543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.828767 (+ 224us) server_negotiation.cc:407] Connection header received
0928 14:51:39.829454 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.829456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.829480 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.829525 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:39.830290 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.830804 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.831815 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.832090 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.832162 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.833153 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:39.833154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:39.833155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:39.833162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:39.833184 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:39.833189 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:39.833254 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.833347 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.833383 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:51:39.833699 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.828377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:40223 (local address 127.18.96.129:50007)
0928 14:51:39.828642 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.828678 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.829343 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.829610 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:39.829618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:39.829627 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:39.830201 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.830206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.830903 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.830904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.831720 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.831721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.832172 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.832174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.832312 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.832910 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:51:39.832911 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:39.832941 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:51:39.832965 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:39.833255 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:39.833262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:39.833266 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.833352 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.833461 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260928 14:51:39.834232 5174 heartbeater.cc:347] Connected to a master server at 127.18.96.190:40223
I20260928 14:51:39.834342 5174 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:39.834579 5174 heartbeater.cc:511] Master 127.18.96.190:40223 requested a full tablet report, sending...
I20260928 14:51:39.835158 4986 ts_manager.cc:194] Registered new tserver with Master: 7bd8df4433ad4c73bcb5b0991e938544 (127.18.96.129:37235)
I20260928 14:51:39.835790 4986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.129:50007
W20260928 14:51:39.914080 5177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:39.914338 5177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:39.914376 5177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:39.915508 5177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:39.915544 5177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:39.915570 5177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:51:39.916965 5177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:39.917167 5177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:39.917409 5177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:39.922523 5182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:39.922569 5177 server_base.cc:1034] running on GCE node
W20260928 14:51:39.922540 5183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:39.922542 5185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:39.922915 5177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:39.923229 5177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:39.924402 5177 hybrid_clock.cc:648] HybridClock initialized: now 1790607099924393 us; error 33 us; skew 500 ppm
I20260928 14:51:39.925783 5177 webserver.cc:460] Webserver started at http://127.18.96.130:36421/ using document root <none> and password file <none>
I20260928 14:51:39.926023 5177 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:39.926167 5177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:39.926314 5177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:39.927446 5177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "f620b3f5fa1d4777b91704f3dd534751"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "9a2489cc316d788b2d54ae9b3d1b7f37"
server_key_iv: "30a5501d30b13d150808106a00f39748"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.927810 5177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "f620b3f5fa1d4777b91704f3dd534751"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "9a2489cc316d788b2d54ae9b3d1b7f37"
server_key_iv: "30a5501d30b13d150808106a00f39748"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.929278 5177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:39.930128 5191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.930315 5177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:39.930418 5177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "f620b3f5fa1d4777b91704f3dd534751"
format_stamp: "Formatted at 2026-09-28 14:51:39 on dist-test-slave-j968"
server_key: "9a2489cc316d788b2d54ae9b3d1b7f37"
server_key_iv: "30a5501d30b13d150808106a00f39748"
server_key_version: "encryptionkey@0"
I20260928 14:51:39.930490 5177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:39.944275 5177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:39.944633 5177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:39.944787 5177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:39.945027 5177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:39.945384 5177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:39.945437 5177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.945477 5177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:39.945514 5177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:39.951705 5177 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:46295
I20260928 14:51:39.951751 5304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:46295 every 8 connection(s)
I20260928 14:51:39.952157 5177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:51:39.955089 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 5177
I20260928 14:51:39.955190 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:51:39.955359 18818 external_mini_cluster.cc:1428] Setting key b00ea3e61b4752a1077e84b11731551d
I20260928 14:51:39.956509 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40223
--builtin_ntp_servers=127.18.96.148:43367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:51:39.958499 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.952939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:53463 (local address 127.18.96.190:40223)
0928 14:51:39.953016 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:39.953030 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:39.953289 (+ 259us) server_negotiation.cc:407] Connection header received
0928 14:51:39.953870 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:39.953872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:39.953884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:39.953953 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:39.954648 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.955347 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.956845 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.957178 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.957253 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.958234 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:39.958235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:39.958235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:39.958239 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:39.958254 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:39.958259 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:39.958305 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:39.958411 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 14:51:39.958438 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:51:39.958829 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:39.952940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:40223 (local address 127.18.96.130:53463)
0928 14:51:39.953159 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:39.953196 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:39.953758 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:39.954032 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:39.954036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:39.954049 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:39.954555 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.954559 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.955495 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.955496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.956760 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:39.956763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:39.957261 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:39.957262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:39.957387 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:39.958052 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:51:39.958052 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:39.958099 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:51:39.958133 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:39.958322 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:39.958328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:39.958332 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:39.958408 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:51:39.958537 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260928 14:51:39.959437 5305 heartbeater.cc:347] Connected to a master server at 127.18.96.190:40223
I20260928 14:51:39.959559 5305 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:39.959754 5305 heartbeater.cc:511] Master 127.18.96.190:40223 requested a full tablet report, sending...
I20260928 14:51:39.960181 4986 ts_manager.cc:194] Registered new tserver with Master: f620b3f5fa1d4777b91704f3dd534751 (127.18.96.130:46295)
I20260928 14:51:39.960598 4986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.130:53463
W20260928 14:51:40.042109 5308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:40.042361 5308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:40.042404 5308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:40.043715 5308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:40.043783 5308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:40.043828 5308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:51:40.045398 5308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:40223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:40.045686 5308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:40.046015 5308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:40.051669 5314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:40.051678 5316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:40.051740 5308 server_base.cc:1034] running on GCE node
W20260928 14:51:40.051695 5313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:40.052104 5308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:40.052378 5308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:40.053527 5308 hybrid_clock.cc:648] HybridClock initialized: now 1790607100053533 us; error 48 us; skew 500 ppm
I20260928 14:51:40.054867 5308 webserver.cc:460] Webserver started at http://127.18.96.131:43999/ using document root <none> and password file <none>
I20260928 14:51:40.055181 5308 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:40.055254 5308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:40.055398 5308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:40.056551 5308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "b2bd4ebfd74144d393e3367620169925"
format_stamp: "Formatted at 2026-09-28 14:51:40 on dist-test-slave-j968"
server_key: "1f0b2e3cbec6d91370696788c4c9e995"
server_key_iv: "d1c8a4305b39d11c611fdf9500607b13"
server_key_version: "encryptionkey@0"
I20260928 14:51:40.056903 5308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "b2bd4ebfd74144d393e3367620169925"
format_stamp: "Formatted at 2026-09-28 14:51:40 on dist-test-slave-j968"
server_key: "1f0b2e3cbec6d91370696788c4c9e995"
server_key_iv: "d1c8a4305b39d11c611fdf9500607b13"
server_key_version: "encryptionkey@0"
I20260928 14:51:40.058212 5308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:51:40.058960 5322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:40.059127 5308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:51:40.059229 5308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "b2bd4ebfd74144d393e3367620169925"
format_stamp: "Formatted at 2026-09-28 14:51:40 on dist-test-slave-j968"
server_key: "1f0b2e3cbec6d91370696788c4c9e995"
server_key_iv: "d1c8a4305b39d11c611fdf9500607b13"
server_key_version: "encryptionkey@0"
I20260928 14:51:40.059301 5308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:40.076464 5308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:40.076799 5308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:40.076947 5308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:40.077195 5308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:40.077543 5308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:40.077595 5308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:40.077638 5308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:40.077664 5308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:40.083407 5308 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:45207
I20260928 14:51:40.083454 5435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:45207 every 8 connection(s)
I20260928 14:51:40.083792 5308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:51:40.089208 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.084574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:37573 (local address 127.18.96.190:40223)
0928 14:51:40.084640 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:40.084642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:40.084880 (+ 238us) server_negotiation.cc:407] Connection header received
0928 14:51:40.085348 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:40.085349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:40.085360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:40.085408 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:40.086190 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.086685 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.087788 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.088053 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.088112 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.088961 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:40.088962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:40.088962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:40.088966 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:40.088979 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:40.088982 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:40.089027 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:40.089119 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 14:51:40.089146 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:51:40.089435 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.084574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:40223 (local address 127.18.96.131:37573)
0928 14:51:40.084758 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:40.084791 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:40.085268 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:40.085469 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:40.085472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:40.085480 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:40.086103 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.086109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.086799 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.086800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.087699 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.087701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.088151 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.088153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.088274 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.088832 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:51:40.088833 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:40.088862 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:51:40.088884 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:40.089038 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:40.089041 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:40.089044 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:51:40.089111 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:51:40.089192 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260928 14:51:40.089941 5436 heartbeater.cc:347] Connected to a master server at 127.18.96.190:40223
I20260928 14:51:40.090040 5436 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:40.090225 5436 heartbeater.cc:511] Master 127.18.96.190:40223 requested a full tablet report, sending...
I20260928 14:51:40.090595 4986 ts_manager.cc:194] Registered new tserver with Master: b2bd4ebfd74144d393e3367620169925 (127.18.96.131:45207)
I20260928 14:51:40.090947 4986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.131:37573
I20260928 14:51:40.092721 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 5308
I20260928 14:51:40.092825 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:51:40.093005 18818 external_mini_cluster.cc:1428] Setting key 3521041694ecf3395a434da2eee3c3bf
I20260928 14:51:40.093873 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:51:40.098536 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.095315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.18.96.190:40223)
0928 14:51:40.095375 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:40.095377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:40.095493 (+ 116us) server_negotiation.cc:407] Connection header received
0928 14:51:40.095549 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:40.095550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:40.095566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:40.095607 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:40.095746 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.096283 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.097114 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.097293 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.097348 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.098206 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:40.098207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:40.098208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:40.098213 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:40.098227 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:40.098231 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:40.098322 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:40.098456 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:51:40.098467 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:51:40.100405 4986 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:51:40.101976 4986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:51:40.108834 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.104741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.18.96.129:37235)
0928 14:51:40.104936 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:40.104941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:40.104952 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:51:40.104995 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:40.104998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:40.105013 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:40.105112 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:40.105528 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.106666 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.107856 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.108387 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.108473 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.108576 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:40.108683 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 14:51:40.108723 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260928 14:51:40.108999 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.104601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:46295 (local address 127.0.0.1:53580)
0928 14:51:40.104925 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:40.104935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:40.104990 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:40.105213 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:40.105217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:40.105219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:40.105288 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.105290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.106665 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.106666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.107696 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.107698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.108703 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.108703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.108753 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.108757 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:40.108803 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 14:51:40.108844 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":6,"thread_start_us":64,"threads_started":1}
I20260928 14:51:40.109071 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.104741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.18.96.130:46295)
0928 14:51:40.104938 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:40.104941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:40.104980 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:51:40.105055 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:40.105057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:40.105070 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:40.105140 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:40.105367 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.106572 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.107789 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.108508 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.108582 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.108669 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:40.108924 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 14:51:40.108976 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260928 14:51:40.109117 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.104511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.18.96.131:45207)
0928 14:51:40.104964 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:40.104968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:40.104978 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:51:40.105020 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:40.105022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:40.105070 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:40.105142 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:40.105378 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.106171 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.107580 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.108361 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.108432 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.108528 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:40.108998 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 14:51:40.109041 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":141,"threads_started":1}
I20260928 14:51:40.108882 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.104384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:45207 (local address 127.0.0.1:54642)
0928 14:51:40.104776 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:40.104788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:40.104844 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:40.105213 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:40.105219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:40.105222 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:40.105297 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.105299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.106264 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.106266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.107485 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.107488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.108565 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.108566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.108626 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.108630 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:40.108685 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:51:40.108731 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":59,"threads_started":1}
I20260928 14:51:40.109479 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.104351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:37235 (local address 127.0.0.1:33582)
0928 14:51:40.104492 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:40.104621 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:40.104701 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:40.105415 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:40.105416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:40.105418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:40.105450 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.105451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.106732 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.106734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.107773 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.107775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.108485 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.108486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.108569 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.108586 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 14:51:40.108670 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:51:40.108728 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":96,"threads_started":1}
I20260928 14:51:40.109786 5239 tablet_service.cc:1467] Processing CreateTablet for tablet a675315cab3b484fbbe190fcf9b84b5e (DEFAULT_TABLE table=test-table [id=6b423642734e470c90f37d8804e3923d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:40.109967 5370 tablet_service.cc:1467] Processing CreateTablet for tablet a675315cab3b484fbbe190fcf9b84b5e (DEFAULT_TABLE table=test-table [id=6b423642734e470c90f37d8804e3923d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:40.110118 5108 tablet_service.cc:1467] Processing CreateTablet for tablet a675315cab3b484fbbe190fcf9b84b5e (DEFAULT_TABLE table=test-table [id=6b423642734e470c90f37d8804e3923d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:40.110224 5239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a675315cab3b484fbbe190fcf9b84b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:40.110314 5370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a675315cab3b484fbbe190fcf9b84b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:40.110410 5108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a675315cab3b484fbbe190fcf9b84b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:40.113296 5450 tablet_bootstrap.cc:492] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751: Bootstrap starting.
I20260928 14:51:40.113686 5451 tablet_bootstrap.cc:492] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: Bootstrap starting.
I20260928 14:51:40.113993 5452 tablet_bootstrap.cc:492] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544: Bootstrap starting.
I20260928 14:51:40.114157 5450 tablet_bootstrap.cc:654] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:40.114326 5451 tablet_bootstrap.cc:654] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:40.114498 5450 log.cc:824] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:40.114634 5451 log.cc:824] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:40.114674 5452 tablet_bootstrap.cc:654] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:40.114940 5452 log.cc:824] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:40.115314 5451 tablet_bootstrap.cc:492] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: No bootstrap required, opened a new log
I20260928 14:51:40.115394 5451 ts_tablet_manager.cc:1397] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:51:40.115655 5452 tablet_bootstrap.cc:492] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544: No bootstrap required, opened a new log
I20260928 14:51:40.115710 5452 ts_tablet_manager.cc:1397] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:51:40.116250 5450 tablet_bootstrap.cc:492] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751: No bootstrap required, opened a new log
I20260928 14:51:40.116330 5450 ts_tablet_manager.cc:1397] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:51:40.117597 5451 raft_consensus.cc:348] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } }
I20260928 14:51:40.117765 5451 raft_consensus.cc:374] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:40.117810 5451 raft_consensus.cc:729] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2bd4ebfd74144d393e3367620169925, State: Initialized, Role: FOLLOWER
I20260928 14:51:40.117950 5451 consensus_queue.cc:260] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } }
I20260928 14:51:40.118252 5436 heartbeater.cc:503] Master 127.18.96.190:40223 was elected leader, sending a full tablet report...
I20260928 14:51:40.118494 5451 ts_tablet_manager.cc:1428] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:51:40.118423 5452 raft_consensus.cc:348] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } }
I20260928 14:51:40.118582 5452 raft_consensus.cc:374] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:40.118618 5452 raft_consensus.cc:729] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd8df4433ad4c73bcb5b0991e938544, State: Initialized, Role: FOLLOWER
I20260928 14:51:40.118754 5452 consensus_queue.cc:260] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } }
I20260928 14:51:40.119014 5452 ts_tablet_manager.cc:1428] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 14:51:40.119026 5450 raft_consensus.cc:348] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } }
I20260928 14:51:40.119232 5174 heartbeater.cc:503] Master 127.18.96.190:40223 was elected leader, sending a full tablet report...
I20260928 14:51:40.119220 5450 raft_consensus.cc:374] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:40.119269 5450 raft_consensus.cc:729] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f620b3f5fa1d4777b91704f3dd534751, State: Initialized, Role: FOLLOWER
I20260928 14:51:40.119381 5450 consensus_queue.cc:260] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } }
I20260928 14:51:40.119638 5450 ts_tablet_manager.cc:1428] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 14:51:40.119683 5305 heartbeater.cc:503] Master 127.18.96.190:40223 was elected leader, sending a full tablet report...
I20260928 14:51:40.189987 5458 raft_consensus.cc:482] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:40.190135 5458 raft_consensus.cc:504] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } }
I20260928 14:51:40.190475 5458 leader_election.cc:288] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bd8df4433ad4c73bcb5b0991e938544 (127.18.96.129:37235), b2bd4ebfd74144d393e3367620169925 (127.18.96.131:45207)
I20260928 14:51:40.194602 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.190657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:45207 (local address 127.18.96.130:46457)
0928 14:51:40.190704 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:40.190719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:40.190788 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:40.191023 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:40.191024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:40.191026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:40.191104 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.191106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.192138 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.192139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.193436 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.193439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.194232 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.194233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.194349 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.194353 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:40.194421 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:51:40.194480 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":31}
I20260928 14:51:40.194617 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.190788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:46457 (local address 127.18.96.131:45207)
0928 14:51:40.190845 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:40.190847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:40.190856 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:51:40.190892 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:40.190893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:40.190904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:40.190980 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:40.191227 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.192045 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.193555 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.194159 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.194228 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.194236 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:40.194467 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 14:51:40.194549 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:51:40.194613 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.190752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:60451 (local address 127.18.96.129:37235)
0928 14:51:40.190829 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:40.190831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:40.191204 (+ 373us) server_negotiation.cc:407] Connection header received
0928 14:51:40.191238 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:40.191239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:40.191251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:40.191311 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:40.191448 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.192203 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.193555 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.194144 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.194228 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.194236 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:40.194476 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 14:51:40.194533 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:51:40.194612 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.190659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:37235 (local address 127.18.96.130:60451)
0928 14:51:40.191111 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:40.191122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:40.191188 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:40.191319 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:40.191319 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:40.191325 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:40.191382 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.191383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.192285 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.192286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.193439 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:40.193440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.194232 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.194233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:40.194349 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.194353 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:40.194421 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:51:40.194479 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":158,"thread_start_us":137,"threads_started":1}
I20260928 14:51:40.195256 5390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a675315cab3b484fbbe190fcf9b84b5e" candidate_uuid: "f620b3f5fa1d4777b91704f3dd534751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2bd4ebfd74144d393e3367620169925" is_pre_election: true
I20260928 14:51:40.195281 5128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a675315cab3b484fbbe190fcf9b84b5e" candidate_uuid: "f620b3f5fa1d4777b91704f3dd534751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd8df4433ad4c73bcb5b0991e938544" is_pre_election: true
I20260928 14:51:40.195420 5390 raft_consensus.cc:2454] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f620b3f5fa1d4777b91704f3dd534751 in term 0.
I20260928 14:51:40.195443 5128 raft_consensus.cc:2454] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f620b3f5fa1d4777b91704f3dd534751 in term 0.
I20260928 14:51:40.195674 5192 leader_election.cc:302] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2bd4ebfd74144d393e3367620169925, f620b3f5fa1d4777b91704f3dd534751; no voters:
I20260928 14:51:40.195828 5458 raft_consensus.cc:2781] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:51:40.195902 5458 raft_consensus.cc:482] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:51:40.195926 5458 raft_consensus.cc:3037] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:40.196692 5458 raft_consensus.cc:504] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } }
I20260928 14:51:40.196849 5458 leader_election.cc:288] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [CANDIDATE]: Term 1 election: Requested vote from peers 7bd8df4433ad4c73bcb5b0991e938544 (127.18.96.129:37235), b2bd4ebfd74144d393e3367620169925 (127.18.96.131:45207)
I20260928 14:51:40.197036 5128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a675315cab3b484fbbe190fcf9b84b5e" candidate_uuid: "f620b3f5fa1d4777b91704f3dd534751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd8df4433ad4c73bcb5b0991e938544"
I20260928 14:51:40.197116 5390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a675315cab3b484fbbe190fcf9b84b5e" candidate_uuid: "f620b3f5fa1d4777b91704f3dd534751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2bd4ebfd74144d393e3367620169925"
I20260928 14:51:40.197122 5128 raft_consensus.cc:3037] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:40.197211 5390 raft_consensus.cc:3037] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:40.198074 5390 raft_consensus.cc:2454] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f620b3f5fa1d4777b91704f3dd534751 in term 1.
I20260928 14:51:40.198273 5128 raft_consensus.cc:2454] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f620b3f5fa1d4777b91704f3dd534751 in term 1.
I20260928 14:51:40.198307 5192 leader_election.cc:302] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2bd4ebfd74144d393e3367620169925, f620b3f5fa1d4777b91704f3dd534751; no voters:
I20260928 14:51:40.198431 5458 raft_consensus.cc:2781] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:40.198596 5458 raft_consensus.cc:686] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [term 1 LEADER]: Becoming Leader. State: Replica: f620b3f5fa1d4777b91704f3dd534751, State: Running, Role: LEADER
I20260928 14:51:40.198705 5458 consensus_queue.cc:237] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } }
I20260928 14:51:40.199572 4985 catalog_manager.cc:5495] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 reported cstate change: term changed from 0 to 1, leader changed from <none> to f620b3f5fa1d4777b91704f3dd534751 (127.18.96.130). New cstate: current_term: 1 leader_uuid: "f620b3f5fa1d4777b91704f3dd534751" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f620b3f5fa1d4777b91704f3dd534751" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 46295 } health_report { overall_health: HEALTHY } } }
W20260928 14:51:40.202682 5306 tablet.cc:2369] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:51:40.225584 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.223142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.18.96.130:46295)
0928 14:51:40.223218 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:40.223220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:40.223227 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:51:40.223263 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:40.223264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:40.223284 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:40.223362 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:51:40.223498 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.224020 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.224777 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:40.224971 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:40.225045 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:40.225206 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:51:40.225298 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:51:40.225407 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:40.225438 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 14:51:40.225446 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:51:40.227524 5390 raft_consensus.cc:1261] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [term 1 FOLLOWER]: Refusing update from remote peer f620b3f5fa1d4777b91704f3dd534751: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:40.227535 5128 raft_consensus.cc:1261] T a675315cab3b484fbbe190fcf9b84b5e P 7bd8df4433ad4c73bcb5b0991e938544 [term 1 FOLLOWER]: Refusing update from remote peer f620b3f5fa1d4777b91704f3dd534751: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:40.227969 5460 consensus_queue.cc:1035] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd8df4433ad4c73bcb5b0991e938544" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:40.228120 5458 consensus_queue.cc:1035] T a675315cab3b484fbbe190fcf9b84b5e P f620b3f5fa1d4777b91704f3dd534751 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2bd4ebfd74144d393e3367620169925" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:40.231217 5465 mvcc.cc:205] Tried to move back new op lower bound from 7334326682528927744 to 7334326682414645248. Current Snapshot: MvccSnapshot[applied={T|T < 7334326682528927744}]
I20260928 14:51:40.233494 4985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:40.233608 4985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:40.234359 4985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d59e333869b84e029bf2b45e49f279b1: Sending DeleteTablet for 3 replicas of tablet a675315cab3b484fbbe190fcf9b84b5e
I20260928 14:51:40.234694 5370 tablet_service.cc:1514] Processing DeleteTablet for tablet a675315cab3b484fbbe190fcf9b84b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:40 UTC) from {username='slave'} at 127.0.0.1:54642
I20260928 14:51:40.234935 5108 tablet_service.cc:1514] Processing DeleteTablet for tablet a675315cab3b484fbbe190fcf9b84b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:40 UTC) from {username='slave'} at 127.0.0.1:33582
I20260928 14:51:40.234937 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 5046
I20260928 14:51:40.235847 5239 tablet_service.cc:1514] Processing DeleteTablet for tablet a675315cab3b484fbbe190fcf9b84b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:40 UTC) from {username='slave'} at 127.0.0.1:53580
I20260928 14:51:40.236486 5472 tablet_replica.cc:330] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: stopping tablet replica
I20260928 14:51:40.236610 5472 raft_consensus.cc:2229] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:51:40.236724 5472 raft_consensus.cc:2258] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:40.237192 5472 ts_tablet_manager.cc:1905] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:40.239289 5472 ts_tablet_manager.cc:1918] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:40.239369 5472 log.cc:1196] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/a675315cab3b484fbbe190fcf9b84b5e
I20260928 14:51:40.239611 5472 ts_tablet_manager.cc:1939] T a675315cab3b484fbbe190fcf9b84b5e P b2bd4ebfd74144d393e3367620169925: Deleting consensus metadata
I20260928 14:51:40.240110 4970 catalog_manager.cc:4841] TS b2bd4ebfd74144d393e3367620169925 (127.18.96.131:45207): tablet a675315cab3b484fbbe190fcf9b84b5e (table test-table [id=6b423642734e470c90f37d8804e3923d]) successfully deleted
W20260928 14:51:40.241513 4973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:37235 (error 108)
I20260928 14:51:40.241909 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 5177
I20260928 14:51:40.242062 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.241892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:37235 (local address 127.0.0.1:33584)
0928 14:51:40.241967 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:40.242025 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:37235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:51:40.243808 4973 catalog_manager.cc:4584] TS 7bd8df4433ad4c73bcb5b0991e938544 (127.18.96.129:37235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a675315cab3b484fbbe190fcf9b84b5e: Network error: Client connection negotiation failed: client connection to 127.18.96.129:37235: connect: Connection refused (error 111)
I20260928 14:51:40.247502 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:40.247353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:46295 (local address 127.0.0.1:53600)
0928 14:51:40.247411 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:40.247462 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.130:46295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:51:40.247563 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 5308
I20260928 14:51:40.252446 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 4955
2026-09-28T14:51:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (779 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:40 dist-test-slave-j968 krb5kdc[5477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:40 dist-test-slave-j968 krb5kdc[5477](info): set up 2 sockets
Sep 28 14:51:40 dist-test-slave-j968 krb5kdc[5477](info): commencing operation
krb5kdc: starting...
W20260928 14:51:41.308869 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:41 dist-test-slave-j968 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:51:41.431598 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:42733
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:42733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:41.523030 5490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:41.523272 5490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:41.523298 5490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:41.524667 5490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:41.524725 5490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:41.524740 5490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:41.524753 5490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:41.524765 5490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:41.526350 5490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:42733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:42733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:41.526705 5490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:41.527034 5490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:41.532451 5498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:41.532451 5496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:41.532516 5495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:41.532829 5490 server_base.cc:1034] running on GCE node
I20260928 14:51:41.533103 5490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:41.533394 5490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:41.534618 5490 hybrid_clock.cc:648] HybridClock initialized: now 1790607101534593 us; error 36 us; skew 500 ppm
Sep 28 14:51:41 dist-test-slave-j968 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607101, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:41.537297 5490 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:51:41.537717 5490 webserver.cc:460] Webserver started at http://127.18.96.190:42267/ using document root <none> and password file <none>
I20260928 14:51:41.537950 5490 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:41.538017 5490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:41.538169 5490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:41.539261 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "8cd2848513d9458bb6f76e52122ddb50"
format_stamp: "Formatted at 2026-09-28 14:51:41 on dist-test-slave-j968"
server_key: "4d14fd68950117ccf1030c8a8be9402e"
server_key_iv: "46416677aedf66934cf32664db01a3fd"
server_key_version: "encryptionkey@0"
I20260928 14:51:41.539637 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "8cd2848513d9458bb6f76e52122ddb50"
format_stamp: "Formatted at 2026-09-28 14:51:41 on dist-test-slave-j968"
server_key: "4d14fd68950117ccf1030c8a8be9402e"
server_key_iv: "46416677aedf66934cf32664db01a3fd"
server_key_version: "encryptionkey@0"
I20260928 14:51:41.541056 5490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:51:41.541868 5505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:41.542110 5490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:41.542212 5490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "8cd2848513d9458bb6f76e52122ddb50"
format_stamp: "Formatted at 2026-09-28 14:51:41 on dist-test-slave-j968"
server_key: "4d14fd68950117ccf1030c8a8be9402e"
server_key_iv: "46416677aedf66934cf32664db01a3fd"
server_key_version: "encryptionkey@0"
I20260928 14:51:41.542304 5490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:41.552623 5490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:41.555263 5490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:41.555430 5490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:41.559671 5490 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:42733
I20260928 14:51:41.559724 5557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:42733 every 8 connection(s)
I20260928 14:51:41.560140 5490 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:41.561113 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:41.563490 5558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50: Bootstrap starting.
I20260928 14:51:41.564183 5558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:41.564455 5558 log.cc:824] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:41.565151 5558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50: No bootstrap required, opened a new log
I20260928 14:51:41.566761 5558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd2848513d9458bb6f76e52122ddb50" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42733 } }
I20260928 14:51:41.566905 5558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:41.566928 5558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd2848513d9458bb6f76e52122ddb50, State: Initialized, Role: FOLLOWER
I20260928 14:51:41.566994 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 5490
I20260928 14:51:41.567099 5558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd2848513d9458bb6f76e52122ddb50" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42733 } }
I20260928 14:51:41.567191 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:41.567199 5558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:41.567233 5558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:41.567286 5558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:41.567406 18818 external_mini_cluster.cc:1428] Setting key 673ed742bf2b3de6db2926a0a1c36a04
I20260928 14:51:41.568053 5558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd2848513d9458bb6f76e52122ddb50" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42733 } }
I20260928 14:51:41.568177 5558 leader_election.cc:302] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd2848513d9458bb6f76e52122ddb50; no voters:
I20260928 14:51:41.568338 5558 leader_election.cc:288] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:41.568367 5563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:41.568470 5563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [term 1 LEADER]: Becoming Leader. State: Replica: 8cd2848513d9458bb6f76e52122ddb50, State: Running, Role: LEADER
I20260928 14:51:41.568559 5563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd2848513d9458bb6f76e52122ddb50" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42733 } }
I20260928 14:51:41.568612 5558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:41.568881 5563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cd2848513d9458bb6f76e52122ddb50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd2848513d9458bb6f76e52122ddb50" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42733 } } }
I20260928 14:51:41.568939 5563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:41.569105 5563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cd2848513d9458bb6f76e52122ddb50. Latest consensus state: current_term: 1 leader_uuid: "8cd2848513d9458bb6f76e52122ddb50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd2848513d9458bb6f76e52122ddb50" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42733 } } }
I20260928 14:51:41.569152 5563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:41.569751 5568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:41.570304 5568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:41.571805 5568 catalog_manager.cc:1347] Generated new cluster ID: d48ca5abdcad4ee1ac7756640e8206a0
I20260928 14:51:41.571858 5568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:51:41 dist-test-slave-j968 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:41.578284 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:41.567919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43836 (local address 127.18.96.190:42733)
0928 14:51:41.568122 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:41.568144 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:41.568181 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:51:41.568309 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:41.568315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:41.568360 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:41.568525 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:41.569291 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.570645 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.571583 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.572202 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.572339 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:41.574997 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:41.575002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:41.575003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:41.575023 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:41.576667 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:41.577118 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:41.577122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:41.577123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:41.577227 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:41.577438 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:41.577439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:41.577439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:41.577700 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:41.577842 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:41.577966 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 14:51:41.578117 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
I20260928 14:51:41.587497 5568 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:41.588268 5568 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:41.596874 5568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50: Generated new TSK 0
I20260928 14:51:41.597148 5568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
Entry for principal kudu/127.18.96.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab.
I20260928 14:51:41.630843 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:42733
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:41.716590 5584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:41.716799 5584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:41.716825 5584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:41.718007 5584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:41.718050 5584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:41.718078 5584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:51:41.719559 5584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.129.keytab
--principal=kudu/127.18.96.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:42733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:41.719817 5584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:41.720077 5584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:41.725142 5590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:41.725365 5584 server_base.cc:1034] running on GCE node
W20260928 14:51:41.725200 5589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:41.725149 5592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:41.725745 5584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:41.725962 5584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:41.727118 5584 hybrid_clock.cc:648] HybridClock initialized: now 1790607101727081 us; error 47 us; skew 500 ppm
Sep 28 14:51:41 dist-test-slave-j968 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607101, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:41.729851 5584 init.cc:377] Logged in from keytab as kudu/127.18.96.129@KRBTEST.COM (short username kudu)
I20260928 14:51:41.730345 5584 webserver.cc:460] Webserver started at http://127.18.96.129:37643/ using document root <none> and password file <none>
I20260928 14:51:41.730595 5584 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:41.730643 5584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:41.730787 5584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:41.731849 5584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "c2a1465b81aa47f398e6ab6c5b493aad"
format_stamp: "Formatted at 2026-09-28 14:51:41 on dist-test-slave-j968"
server_key: "f63706a32ce1b2ef2bc84baa54b13efd"
server_key_iv: "b409086a45ee25a0832805d325e2455e"
server_key_version: "encryptionkey@0"
I20260928 14:51:41.732291 5584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "c2a1465b81aa47f398e6ab6c5b493aad"
format_stamp: "Formatted at 2026-09-28 14:51:41 on dist-test-slave-j968"
server_key: "f63706a32ce1b2ef2bc84baa54b13efd"
server_key_iv: "b409086a45ee25a0832805d325e2455e"
server_key_version: "encryptionkey@0"
I20260928 14:51:41.733659 5584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:41.734508 5599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:41.734714 5584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:41.734808 5584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "c2a1465b81aa47f398e6ab6c5b493aad"
format_stamp: "Formatted at 2026-09-28 14:51:41 on dist-test-slave-j968"
server_key: "f63706a32ce1b2ef2bc84baa54b13efd"
server_key_iv: "b409086a45ee25a0832805d325e2455e"
server_key_version: "encryptionkey@0"
I20260928 14:51:41.734879 5584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:41.747561 5584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:41.749959 5584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:41.750133 5584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:41.750378 5584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:41.750700 5584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:41.750734 5584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:41.750757 5584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:41.750770 5584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:41.756948 5584 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:46601
I20260928 14:51:41.756989 5712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:46601 every 8 connection(s)
I20260928 14:51:41.757351 5584 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:51:41 dist-test-slave-j968 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607101, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.129@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:41.765965 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 5584
I20260928 14:51:41.766094 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:51:41.766320 18818 external_mini_cluster.cc:1428] Setting key dc1d2c8906cb98c501e261807e9b14d7
I20260928 14:51:41.767913 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:41.758184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:43127 (local address 127.18.96.190:42733)
0928 14:51:41.758261 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:41.758263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:41.758457 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:51:41.759508 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:41.759510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:41.759529 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:41.759598 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:41.760366 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.760983 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.762121 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.762335 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.762399 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:41.764932 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:41.764933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:41.764933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:41.764942 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:41.766271 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:41.766733 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:41.766735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:41.766735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:41.766794 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:41.767136 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:41.767136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:41.767137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:41.767305 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:41.767365 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:41.767720 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 14:51:41.767801 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:51:41.768249 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:41.758133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42733 (local address 127.18.96.129:43127)
0928 14:51:41.758295 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:41.758340 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:41.759385 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:41.759665 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:41.759669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:41.760029 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:41.760251 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:41.760257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.761103 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.761104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:41.761941 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:41.761944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.762444 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.762445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:41.762577 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:41.763258 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:41.763267 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:41.764809 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:41.766351 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:41.766361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:41.766364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:41.766661 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:41.766857 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:41.766858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:41.766859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:41.767005 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:41.767375 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:41.767383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:41.767566 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 14:51:41.767692 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:51:41.767924 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260928 14:51:41.769079 5713 heartbeater.cc:347] Connected to a master server at 127.18.96.190:42733
I20260928 14:51:41.769207 5713 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:41.769421 5713 heartbeater.cc:511] Master 127.18.96.190:42733 requested a full tablet report, sending...
I20260928 14:51:41.769979 5522 ts_manager.cc:194] Registered new tserver with Master: c2a1465b81aa47f398e6ab6c5b493aad (127.18.96.129:46601)
I20260928 14:51:41.770694 5522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.129@KRBTEST.COM'} at 127.18.96.129:43127
WARNING: no policy specified for kudu/127.18.96.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
Entry for principal kudu/127.18.96.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab.
I20260928 14:51:41.794997 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:42733
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:41.879323 5718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:41.879532 5718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:41.879556 5718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:41.880800 5718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:41.880843 5718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:41.880870 5718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:51:41.882347 5718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.130.keytab
--principal=kudu/127.18.96.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:42733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:41.882586 5718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:41.882830 5718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:41.887959 5724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:41.887995 5718 server_base.cc:1034] running on GCE node
W20260928 14:51:41.887969 5726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:41.888007 5723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:41.888378 5718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:41.888624 5718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:41.889784 5718 hybrid_clock.cc:648] HybridClock initialized: now 1790607101889764 us; error 37 us; skew 500 ppm
Sep 28 14:51:41 dist-test-slave-j968 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607101, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:41.892290 5718 init.cc:377] Logged in from keytab as kudu/127.18.96.130@KRBTEST.COM (short username kudu)
I20260928 14:51:41.892748 5718 webserver.cc:460] Webserver started at http://127.18.96.130:42845/ using document root <none> and password file <none>
I20260928 14:51:41.892972 5718 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:41.893019 5718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:41.893147 5718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:41.894210 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "e47a63f9e171429e8735bb859381dae4"
format_stamp: "Formatted at 2026-09-28 14:51:41 on dist-test-slave-j968"
server_key: "17e29f74c862ac3b9c01362c086c6d38"
server_key_iv: "2ea4ca0c04ef66eada8a5534ff0c0a2e"
server_key_version: "encryptionkey@0"
I20260928 14:51:41.894569 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "e47a63f9e171429e8735bb859381dae4"
format_stamp: "Formatted at 2026-09-28 14:51:41 on dist-test-slave-j968"
server_key: "17e29f74c862ac3b9c01362c086c6d38"
server_key_iv: "2ea4ca0c04ef66eada8a5534ff0c0a2e"
server_key_version: "encryptionkey@0"
I20260928 14:51:41.895968 5718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 14:51:41.896778 5733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:41.897014 5718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:41.897120 5718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "e47a63f9e171429e8735bb859381dae4"
format_stamp: "Formatted at 2026-09-28 14:51:41 on dist-test-slave-j968"
server_key: "17e29f74c862ac3b9c01362c086c6d38"
server_key_iv: "2ea4ca0c04ef66eada8a5534ff0c0a2e"
server_key_version: "encryptionkey@0"
I20260928 14:51:41.897204 5718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:41.912442 5718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:41.914702 5718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:41.914891 5718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:41.915156 5718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:41.915477 5718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:41.915527 5718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:41.915575 5718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:41.915601 5718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:41.921774 5718 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:42709
I20260928 14:51:41.921816 5846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:42709 every 8 connection(s)
I20260928 14:51:41.922168 5718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:51:41 dist-test-slave-j968 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607101, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.130@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:41.930866 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 5718
I20260928 14:51:41.931113 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:51:41.931406 18818 external_mini_cluster.cc:1428] Setting key 3dc8b55ee2488611b62b1c0622464712
I20260928 14:51:41.932212 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:41.922829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:35221 (local address 127.18.96.190:42733)
0928 14:51:41.922916 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:41.922919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:41.923118 (+ 199us) server_negotiation.cc:407] Connection header received
0928 14:51:41.923634 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:41.923635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:41.923646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:41.923700 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:41.924749 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.925275 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.926326 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.926515 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.926579 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:41.928781 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:41.928783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:41.928783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:41.928791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:41.930269 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:41.930885 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:41.930887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:41.930887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:41.930962 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:41.931336 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:41.931337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:41.931338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:41.931521 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:41.931603 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:41.932016 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 14:51:41.932112 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:51:41.932446 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:41.922816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42733 (local address 127.18.96.130:35221)
0928 14:51:41.922994 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:41.923031 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:41.923511 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:41.923757 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:41.923761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:41.924424 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:41.924643 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:41.924648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.925382 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.925383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:41.926242 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:41.926245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:41.926603 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:41.926604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:41.926712 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:41.927215 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:41.927223 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:41.928689 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:41.930532 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:41.930540 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:41.930542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:41.930805 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:41.931070 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:41.931071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:41.931072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:41.931251 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:41.931622 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:41.931629 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:41.931821 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 14:51:41.931980 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 14:51:41.932173 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260928 14:51:41.932940 5847 heartbeater.cc:347] Connected to a master server at 127.18.96.190:42733
I20260928 14:51:41.933058 5847 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:41.933245 5847 heartbeater.cc:511] Master 127.18.96.190:42733 requested a full tablet report, sending...
I20260928 14:51:41.933652 5522 ts_manager.cc:194] Registered new tserver with Master: e47a63f9e171429e8735bb859381dae4 (127.18.96.130:42709)
I20260928 14:51:41.934038 5522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.130@KRBTEST.COM'} at 127.18.96.130:35221
WARNING: no policy specified for kudu/127.18.96.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
Entry for principal kudu/127.18.96.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab.
I20260928 14:51:41.960772 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:42733
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:42.042721 5852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:42.042929 5852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:42.042954 5852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:42.044168 5852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:42.044210 5852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:42.044237 5852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:51:42.045842 5852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.131.keytab
--principal=kudu/127.18.96.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:42733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:42.046095 5852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:42.046337 5852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:42.051604 5857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:42.051648 5852 server_base.cc:1034] running on GCE node
W20260928 14:51:42.051614 5860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:42.051614 5858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:42.052016 5852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:42.052281 5852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:42.053460 5852 hybrid_clock.cc:648] HybridClock initialized: now 1790607102053424 us; error 38 us; skew 500 ppm
Sep 28 14:51:42 dist-test-slave-j968 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607102, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:42.056352 5852 init.cc:377] Logged in from keytab as kudu/127.18.96.131@KRBTEST.COM (short username kudu)
I20260928 14:51:42.056844 5852 webserver.cc:460] Webserver started at http://127.18.96.131:35069/ using document root <none> and password file <none>
I20260928 14:51:42.057107 5852 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:42.057180 5852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:42.057332 5852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:42.058501 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "658ed3798f264f2b9ebfde9135819f62"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "8f9752f91762f721a5e2f0dc42e97f83"
server_key_iv: "d9b1e3eb847b3b4a45161f3061827392"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.058882 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "658ed3798f264f2b9ebfde9135819f62"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "8f9752f91762f721a5e2f0dc42e97f83"
server_key_iv: "d9b1e3eb847b3b4a45161f3061827392"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.060418 5852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:51:42.061301 5867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.061515 5852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:42.061628 5852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "658ed3798f264f2b9ebfde9135819f62"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "8f9752f91762f721a5e2f0dc42e97f83"
server_key_iv: "d9b1e3eb847b3b4a45161f3061827392"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.061707 5852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:42.094172 5852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:42.096884 5852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:42.097070 5852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:42.097329 5852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:42.097674 5852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:42.097728 5852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.097767 5852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:42.097815 5852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.104449 5852 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:46323
I20260928 14:51:42.104496 5980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:46323 every 8 connection(s)
I20260928 14:51:42.104862 5852 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:51:42.106768 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 5852
I20260928 14:51:42.106889 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:51:42.107089 18818 external_mini_cluster.cc:1428] Setting key a5bd78d33d48dd0b8fc8daf668c355a9
Sep 28 14:51:42 dist-test-slave-j968 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607102, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.131@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:51:42.116600 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.105796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:54203 (local address 127.18.96.190:42733)
0928 14:51:42.105881 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.105884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.106119 (+ 235us) server_negotiation.cc:407] Connection header received
0928 14:51:42.106689 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.106691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.106708 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.106797 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:42.108282 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.108884 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.110346 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.110580 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.110645 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.113172 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:42.113174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:42.113175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:42.113183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:42.114927 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:42.115462 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:42.115464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:42.115465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:42.115530 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:42.115810 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:42.115811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:42.115811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:42.116081 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:42.116149 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.116443 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.116529 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:51:42.116771 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.105773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:42733 (local address 127.18.96.131:54203)
0928 14:51:42.105966 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:42.106005 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:42.106531 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:42.106897 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:42.106903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:42.107820 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:42.108185 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.108193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.109051 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.109052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.109929 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.109932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.110680 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.110681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.110811 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.111427 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:51:42.111436 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:42.113009 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:42.115095 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:42.115103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:42.115105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:42.115375 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:42.115603 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:51:42.115603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:51:42.115604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:51:42.115725 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:51:42.116155 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:42.116160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:42.116298 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 14:51:42.116390 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:51:42.116537 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260928 14:51:42.117388 5981 heartbeater.cc:347] Connected to a master server at 127.18.96.190:42733
I20260928 14:51:42.117511 5981 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:42.117748 5981 heartbeater.cc:511] Master 127.18.96.190:42733 requested a full tablet report, sending...
I20260928 14:51:42.118285 5522 ts_manager.cc:194] Registered new tserver with Master: 658ed3798f264f2b9ebfde9135819f62 (127.18.96.131:46323)
I20260928 14:51:42.118886 5522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.131@KRBTEST.COM'} at 127.18.96.131:54203
I20260928 14:51:42.119094 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:51:42.126732 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.120599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.18.96.190:42733)
0928 14:51:42.120668 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.120670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.120751 (+ 81us) server_negotiation.cc:407] Connection header received
0928 14:51:42.120820 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.120821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.120831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.120890 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:42.121461 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.121976 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.122719 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.122920 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.122990 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.124339 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:42.124341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:42.124341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:42.124349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:42.125558 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:42.125887 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:42.125888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:42.125889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:42.125949 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:42.126144 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:42.126145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:42.126145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:42.126297 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:42.126406 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.126518 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.126631 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:51:42.128537 5522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:51:42.130312 5522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:51:42.137912 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.132961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:46323 (local address 127.0.0.1:35958)
0928 14:51:42.133174 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:42.133220 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:42.133285 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:42.133621 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:42.133625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:42.133627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:42.133708 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.133710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.135155 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.135156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.136532 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.136535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.137573 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.137574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.137662 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.137678 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 14:51:42.137730 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:51:42.137828 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":31,"thread_start_us":47,"threads_started":1}
I20260928 14:51:42.138051 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.133096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.18.96.131:46323)
0928 14:51:42.133426 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.133432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.133442 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:51:42.133493 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.133496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.133514 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.133584 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:42.133802 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.135032 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.136621 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.137438 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.137807 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.137883 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.137911 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.137951 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":157,"threads_started":1}
I20260928 14:51:42.138779 5915 tablet_service.cc:1467] Processing CreateTablet for tablet 6aca080eed5f46e99fc1374734b72de1 (DEFAULT_TABLE table=test-table [id=be2fb6b18ae049c697a5bf319fd6503f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:42.139175 5915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aca080eed5f46e99fc1374734b72de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:42.140062 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.133019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:42709 (local address 127.0.0.1:47956)
0928 14:51:42.135967 (+ 2948us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:42.135978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:42.136029 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:42.136255 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:42.136256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:42.136258 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:42.136343 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.136345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.137662 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.137663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.139131 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.139134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.139773 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.139774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.139835 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.139839 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:42.139930 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:51:42.139970 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2935,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260928 14:51:42.140072 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.133283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.18.96.130:42709)
0928 14:51:42.133478 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.133482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.136093 (+ 2611us) server_negotiation.cc:407] Connection header received
0928 14:51:42.136141 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.136143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.136161 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.136232 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:42.136452 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.137538 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.139199 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.139684 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.139773 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.139837 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.139946 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.139985 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260928 14:51:42.140787 5781 tablet_service.cc:1467] Processing CreateTablet for tablet 6aca080eed5f46e99fc1374734b72de1 (DEFAULT_TABLE table=test-table [id=be2fb6b18ae049c697a5bf319fd6503f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:42.141160 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aca080eed5f46e99fc1374734b72de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:42.142767 5998 tablet_bootstrap.cc:492] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: Bootstrap starting.
I20260928 14:51:42.143410 5998 tablet_bootstrap.cc:654] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:42.143958 5998 log.cc:824] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:42.144312 5999 tablet_bootstrap.cc:492] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: Bootstrap starting.
I20260928 14:51:42.144898 5999 tablet_bootstrap.cc:654] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:42.145553 5999 log.cc:824] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:42.145946 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.135804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:46601 (local address 127.0.0.1:36566)
0928 14:51:42.137015 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:42.137025 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:42.137148 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:42.137489 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:42.137490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:42.137492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:42.137571 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.137573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.139941 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.139942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.144769 (+ 4827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.144773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.145668 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.145670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.145734 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.145738 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:42.145791 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 14:51:42.145840 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"thread_start_us":110,"threads_started":1}
I20260928 14:51:42.146220 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.135941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.18.96.129:46601)
0928 14:51:42.136370 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.136374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.137116 (+ 742us) server_negotiation.cc:407] Connection header received
0928 14:51:42.137313 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.137316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.137332 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.137427 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:42.137767 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.139787 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.144881 (+ 5094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.145562 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.145656 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.145718 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.146051 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.146141 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":162,"threads_started":1}
I20260928 14:51:42.146343 5999 tablet_bootstrap.cc:492] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: No bootstrap required, opened a new log
I20260928 14:51:42.146420 5999 ts_tablet_manager.cc:1397] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:51:42.146724 5998 tablet_bootstrap.cc:492] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: No bootstrap required, opened a new log
I20260928 14:51:42.146797 5998 ts_tablet_manager.cc:1397] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 14:51:42.146930 5647 tablet_service.cc:1467] Processing CreateTablet for tablet 6aca080eed5f46e99fc1374734b72de1 (DEFAULT_TABLE table=test-table [id=be2fb6b18ae049c697a5bf319fd6503f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:42.147244 5647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aca080eed5f46e99fc1374734b72de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:42.148594 5999 raft_consensus.cc:348] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } }
I20260928 14:51:42.148789 5999 raft_consensus.cc:374] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:42.148828 5999 raft_consensus.cc:729] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47a63f9e171429e8735bb859381dae4, State: Initialized, Role: FOLLOWER
I20260928 14:51:42.148845 5998 raft_consensus.cc:348] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } }
I20260928 14:51:42.148958 5999 consensus_queue.cc:260] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } }
I20260928 14:51:42.149036 5998 raft_consensus.cc:374] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:42.149098 5998 raft_consensus.cc:729] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658ed3798f264f2b9ebfde9135819f62, State: Initialized, Role: FOLLOWER
I20260928 14:51:42.149238 5999 ts_tablet_manager.cc:1428] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:51:42.149212 5998 consensus_queue.cc:260] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } }
I20260928 14:51:42.149367 5847 heartbeater.cc:503] Master 127.18.96.190:42733 was elected leader, sending a full tablet report...
I20260928 14:51:42.149469 5998 ts_tablet_manager.cc:1428] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 14:51:42.149652 5981 heartbeater.cc:503] Master 127.18.96.190:42733 was elected leader, sending a full tablet report...
I20260928 14:51:42.150643 6004 tablet_bootstrap.cc:492] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad: Bootstrap starting.
I20260928 14:51:42.151274 6004 tablet_bootstrap.cc:654] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:42.151754 6004 log.cc:824] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:42.152431 6004 tablet_bootstrap.cc:492] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad: No bootstrap required, opened a new log
I20260928 14:51:42.152508 6004 ts_tablet_manager.cc:1397] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:51:42.153977 6004 raft_consensus.cc:348] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } }
I20260928 14:51:42.154155 6004 raft_consensus.cc:374] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:42.154199 6004 raft_consensus.cc:729] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a1465b81aa47f398e6ab6c5b493aad, State: Initialized, Role: FOLLOWER
I20260928 14:51:42.154346 6004 consensus_queue.cc:260] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } }
I20260928 14:51:42.154605 6004 ts_tablet_manager.cc:1428] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:51:42.154661 5713 heartbeater.cc:503] Master 127.18.96.190:42733 was elected leader, sending a full tablet report...
W20260928 14:51:42.172684 5848 tablet.cc:2369] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:51:42.258213 5714 tablet.cc:2369] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:51:42.355683 5982 tablet.cc:2369] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:51:42.377416 6003 raft_consensus.cc:482] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:42.377549 6003 raft_consensus.cc:504] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } }
I20260928 14:51:42.377885 6003 leader_election.cc:288] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a1465b81aa47f398e6ab6c5b493aad (127.18.96.129:46601), e47a63f9e171429e8735bb859381dae4 (127.18.96.130:42709)
I20260928 14:51:42.381199 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.378157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:46601 (local address 127.18.96.131:49981)
0928 14:51:42.378416 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:42.378427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:42.378487 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:42.378645 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:42.378646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:42.378648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:42.378727 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.378729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.379447 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.379447 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.380301 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.380303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.380934 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.380935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.381008 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.381012 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:42.381067 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:51:42.381114 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":64,"thread_start_us":123,"threads_started":1}
I20260928 14:51:42.381237 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.378209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:49981 (local address 127.18.96.129:46601)
0928 14:51:42.378256 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.378258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.378501 (+ 243us) server_negotiation.cc:407] Connection header received
0928 14:51:42.378537 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.378539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.378555 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.378624 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:42.378789 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.379371 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.380395 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.380856 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.380919 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.380926 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.381106 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.381154 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 14:51:42.381423 5983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.378156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:42709 (local address 127.18.96.131:34193)
0928 14:51:42.378209 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:42.378222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:42.378273 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:42.378494 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:42.378496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:42.378498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:42.378583 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.378585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.379291 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.379292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.380425 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.380428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.381107 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.381108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.381201 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.381206 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:51:42.381298 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:51:42.381348 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":82}
I20260928 14:51:42.381551 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.378209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:34193 (local address 127.18.96.130:42709)
0928 14:51:42.378256 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.378258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.378271 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:51:42.378342 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.378354 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.378367 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.378454 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:42.378669 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.379203 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.380558 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.380975 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.381062 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.381072 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.381430 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.381474 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:51:42.381836 5667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aca080eed5f46e99fc1374734b72de1" candidate_uuid: "658ed3798f264f2b9ebfde9135819f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" is_pre_election: true
I20260928 14:51:42.381855 5801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aca080eed5f46e99fc1374734b72de1" candidate_uuid: "658ed3798f264f2b9ebfde9135819f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47a63f9e171429e8735bb859381dae4" is_pre_election: true
I20260928 14:51:42.382050 5667 raft_consensus.cc:2454] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658ed3798f264f2b9ebfde9135819f62 in term 0.
I20260928 14:51:42.382157 5801 raft_consensus.cc:2454] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658ed3798f264f2b9ebfde9135819f62 in term 0.
I20260928 14:51:42.382287 5871 leader_election.cc:302] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658ed3798f264f2b9ebfde9135819f62, c2a1465b81aa47f398e6ab6c5b493aad; no voters:
I20260928 14:51:42.382422 6003 raft_consensus.cc:2781] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:51:42.382522 6003 raft_consensus.cc:482] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:51:42.382544 6003 raft_consensus.cc:3037] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:42.383338 6003 raft_consensus.cc:504] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } }
I20260928 14:51:42.383507 6003 leader_election.cc:288] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [CANDIDATE]: Term 1 election: Requested vote from peers c2a1465b81aa47f398e6ab6c5b493aad (127.18.96.129:46601), e47a63f9e171429e8735bb859381dae4 (127.18.96.130:42709)
I20260928 14:51:42.383684 5667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aca080eed5f46e99fc1374734b72de1" candidate_uuid: "658ed3798f264f2b9ebfde9135819f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a1465b81aa47f398e6ab6c5b493aad"
I20260928 14:51:42.383733 5801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aca080eed5f46e99fc1374734b72de1" candidate_uuid: "658ed3798f264f2b9ebfde9135819f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47a63f9e171429e8735bb859381dae4"
I20260928 14:51:42.383785 5667 raft_consensus.cc:3037] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:42.383817 5801 raft_consensus.cc:3037] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:42.384618 5801 raft_consensus.cc:2454] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658ed3798f264f2b9ebfde9135819f62 in term 1.
I20260928 14:51:42.384636 5667 raft_consensus.cc:2454] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658ed3798f264f2b9ebfde9135819f62 in term 1.
I20260928 14:51:42.384797 5869 leader_election.cc:302] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658ed3798f264f2b9ebfde9135819f62, e47a63f9e171429e8735bb859381dae4; no voters:
I20260928 14:51:42.384907 6003 raft_consensus.cc:2781] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:42.385104 6003 raft_consensus.cc:686] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 1 LEADER]: Becoming Leader. State: Replica: 658ed3798f264f2b9ebfde9135819f62, State: Running, Role: LEADER
I20260928 14:51:42.385211 6003 consensus_queue.cc:237] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } }
I20260928 14:51:42.386083 5522 catalog_manager.cc:5495] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 658ed3798f264f2b9ebfde9135819f62 (127.18.96.131). New cstate: current_term: 1 leader_uuid: "658ed3798f264f2b9ebfde9135819f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658ed3798f264f2b9ebfde9135819f62" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 46323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 } health_report { overall_health: UNKNOWN } } }
I20260928 14:51:42.424180 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.419784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.18.96.131:46323)
0928 14:51:42.419844 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.419846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.419856 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:51:42.420183 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.420184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.420215 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.420328 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:51:42.420595 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.421762 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.422985 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.423328 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.423398 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.423668 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:51:42.423841 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:51:42.423969 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.424034 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.424064 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:51:42.426759 5667 raft_consensus.cc:1261] T 6aca080eed5f46e99fc1374734b72de1 P c2a1465b81aa47f398e6ab6c5b493aad [term 1 FOLLOWER]: Refusing update from remote peer 658ed3798f264f2b9ebfde9135819f62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:42.427171 5801 raft_consensus.cc:1261] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [term 1 FOLLOWER]: Refusing update from remote peer 658ed3798f264f2b9ebfde9135819f62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:42.427156 6003 consensus_queue.cc:1035] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a1465b81aa47f398e6ab6c5b493aad" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 46601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:42.427590 6003 consensus_queue.cc:1035] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47a63f9e171429e8735bb859381dae4" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:42.430047 6011 mvcc.cc:205] Tried to move back new op lower bound from 7334326691536850944 to 7334326691370450944. Current Snapshot: MvccSnapshot[applied={T|T < 7334326691536850944}]
I20260928 14:51:42.430342 6012 mvcc.cc:205] Tried to move back new op lower bound from 7334326691536850944 to 7334326691370450944. Current Snapshot: MvccSnapshot[applied={T|T < 7334326691536850944}]
I20260928 14:51:42.432427 6014 mvcc.cc:205] Tried to move back new op lower bound from 7334326691536850944 to 7334326691370450944. Current Snapshot: MvccSnapshot[applied={T|T < 7334326691536850944 or (T in {7334326691536850944})}]
I20260928 14:51:42.432798 5522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:42.432901 5522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:42.433645 5522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cd2848513d9458bb6f76e52122ddb50: Sending DeleteTablet for 3 replicas of tablet 6aca080eed5f46e99fc1374734b72de1
I20260928 14:51:42.433950 5915 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aca080eed5f46e99fc1374734b72de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:42 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:35958
I20260928 14:51:42.433991 5647 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aca080eed5f46e99fc1374734b72de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:42 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:36566
I20260928 14:51:42.434008 5781 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aca080eed5f46e99fc1374734b72de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:42 UTC) from {username='kudu', principal='kudu/127.18.96.190@KRBTEST.COM'} at 127.0.0.1:47956
I20260928 14:51:42.434257 6022 tablet_replica.cc:330] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: stopping tablet replica
I20260928 14:51:42.434370 6022 raft_consensus.cc:2229] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:51:42.434441 6021 tablet_replica.cc:330] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: stopping tablet replica
I20260928 14:51:42.434535 6022 raft_consensus.cc:2258] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:42.434562 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 5584
I20260928 14:51:42.434597 6021 raft_consensus.cc:2229] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:51:42.434734 6021 raft_consensus.cc:2258] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:42.435249 6021 ts_tablet_manager.cc:1905] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:42.435855 6022 ts_tablet_manager.cc:1905] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:42.437173 6021 ts_tablet_manager.cc:1918] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:42.437311 6021 log.cc:1196] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/6aca080eed5f46e99fc1374734b72de1
I20260928 14:51:42.437654 6021 ts_tablet_manager.cc:1939] T 6aca080eed5f46e99fc1374734b72de1 P e47a63f9e171429e8735bb859381dae4: Deleting consensus metadata
I20260928 14:51:42.438001 6022 ts_tablet_manager.cc:1918] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:42.438131 6022 log.cc:1196] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/6aca080eed5f46e99fc1374734b72de1
I20260928 14:51:42.438118 5507 catalog_manager.cc:4841] TS e47a63f9e171429e8735bb859381dae4 (127.18.96.130:42709): tablet 6aca080eed5f46e99fc1374734b72de1 (table test-table [id=be2fb6b18ae049c697a5bf319fd6503f]) successfully deleted
I20260928 14:51:42.438468 6022 ts_tablet_manager.cc:1939] T 6aca080eed5f46e99fc1374734b72de1 P 658ed3798f264f2b9ebfde9135819f62: Deleting consensus metadata
I20260928 14:51:42.438977 5508 catalog_manager.cc:4841] TS 658ed3798f264f2b9ebfde9135819f62 (127.18.96.131:46323): tablet 6aca080eed5f46e99fc1374734b72de1 (table test-table [id=be2fb6b18ae049c697a5bf319fd6503f]) successfully deleted
I20260928 14:51:42.445528 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 5718
W20260928 14:51:42.453088 5509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:46601 (error 108)
I20260928 14:51:42.453691 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.453541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:46601 (local address 127.0.0.1:36574)
0928 14:51:42.453604 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:42.453644 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:46601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:51:42.454200 5509 catalog_manager.cc:4584] TS c2a1465b81aa47f398e6ab6c5b493aad (127.18.96.129:46601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6aca080eed5f46e99fc1374734b72de1: Network error: Client connection negotiation failed: client connection to 127.18.96.129:46601: connect: Connection refused (error 111)
I20260928 14:51:42.454820 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 5852
I20260928 14:51:42.460916 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 5490
2026-09-28T14:51:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2210 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T14:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:42Z Disabled control of system clock
I20260928 14:51:42.489888 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35689
--webserver_interface=127.18.96.190
--webserver_port=0
--builtin_ntp_servers=127.18.96.148:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:35689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:51:42.574397 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:42.574661 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:42.574685 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:42.576120 6027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:42.576174 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:42.576188 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:42.576203 6027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:42.576217 6027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:42.577760 6027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:35689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:35689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:42.578008 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:42.578301 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:42.583947 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:42.583925 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:42.583961 6027 server_base.cc:1034] running on GCE node
W20260928 14:51:42.583900 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:42.584422 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:42.584766 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:42.585918 6027 hybrid_clock.cc:648] HybridClock initialized: now 1790607102585867 us; error 62 us; skew 500 ppm
I20260928 14:51:42.594545 6027 webserver.cc:460] Webserver started at http://127.18.96.190:35567/ using document root <none> and password file <none>
I20260928 14:51:42.594920 6027 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:42.594975 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:42.595175 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:42.596438 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "8b7b48bee35d4185b23c97ff2d632630"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "bdab1b4e0bde899f6434d9ba5b3a4f40"
server_key_iv: "695f2d7857205eb70f9c21ada441090e"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.596805 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "8b7b48bee35d4185b23c97ff2d632630"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "bdab1b4e0bde899f6434d9ba5b3a4f40"
server_key_iv: "695f2d7857205eb70f9c21ada441090e"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.598228 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:51:42.599155 6041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.599550 6027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:42.599686 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "8b7b48bee35d4185b23c97ff2d632630"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "bdab1b4e0bde899f6434d9ba5b3a4f40"
server_key_iv: "695f2d7857205eb70f9c21ada441090e"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.599767 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:42.608949 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:42.609376 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:42.609532 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:42.613860 6027 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:35689
I20260928 14:51:42.613901 6093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:35689 every 8 connection(s)
I20260928 14:51:42.614351 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:42.615388 6094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:42.617822 6094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630: Bootstrap starting.
I20260928 14:51:42.618436 6094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:42.618737 6094 log.cc:824] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:42.619441 6094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630: No bootstrap required, opened a new log
I20260928 14:51:42.621153 6094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b48bee35d4185b23c97ff2d632630" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35689 } }
I20260928 14:51:42.621300 6094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:42.621322 6094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7b48bee35d4185b23c97ff2d632630, State: Initialized, Role: FOLLOWER
I20260928 14:51:42.621501 6094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b48bee35d4185b23c97ff2d632630" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35689 } }
I20260928 14:51:42.621567 6094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:42.621608 6094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:42.621659 6094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:42.622190 6094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b48bee35d4185b23c97ff2d632630" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35689 } }
I20260928 14:51:42.622300 6094 leader_election.cc:302] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7b48bee35d4185b23c97ff2d632630; no voters:
I20260928 14:51:42.622486 6094 leader_election.cc:288] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:42.622532 6097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:42.622756 6094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:42.622756 6097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7b48bee35d4185b23c97ff2d632630, State: Running, Role: LEADER
I20260928 14:51:42.622959 6097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b48bee35d4185b23c97ff2d632630" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35689 } }
I20260928 14:51:42.623418 6098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b7b48bee35d4185b23c97ff2d632630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b48bee35d4185b23c97ff2d632630" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35689 } } }
I20260928 14:51:42.623503 6098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:42.623597 6099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7b48bee35d4185b23c97ff2d632630. Latest consensus state: current_term: 1 leader_uuid: "8b7b48bee35d4185b23c97ff2d632630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b48bee35d4185b23c97ff2d632630" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 35689 } } }
I20260928 14:51:42.623726 6099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:42.623852 6106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:42.624408 6106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:42.624559 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 6027
I20260928 14:51:42.624668 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:42.624858 18818 external_mini_cluster.cc:1428] Setting key 9781316421f4a3b54e1ef3907110656a
I20260928 14:51:42.628314 6106 catalog_manager.cc:1347] Generated new cluster ID: c8d7c4c2a82f4533a8bd9f3f722ed880
I20260928 14:51:42.628376 6106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:42.630060 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.625387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.18.96.190:35689)
0928 14:51:42.625661 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.625671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.625707 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:51:42.625829 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.625835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.625886 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.626030 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:42.626227 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.627122 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.627906 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.628466 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.628593 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.629351 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:42.629355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:42.629356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:42.629560 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:42.629602 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:42.629616 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:42.629682 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.629774 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.629850 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260928 14:51:42.639606 6106 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:42.640385 6106 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:42.655303 6106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630: Generated new TSK 0
I20260928 14:51:42.655550 6106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:42.657492 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.129:0
--local_ip_for_outbound_sockets=127.18.96.129
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35689
--builtin_ntp_servers=127.18.96.148:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:51:42.743203 6118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:42.743424 6118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:42.743448 6118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:42.744691 6118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:42.744733 6118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:42.744760 6118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.129
I20260928 14:51:42.746165 6118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.129
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.129
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:42.746417 6118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:42.746717 6118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:42.751817 6124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:42.751848 6123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:42.751823 6126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:42.752074 6118 server_base.cc:1034] running on GCE node
I20260928 14:51:42.752393 6118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:42.752674 6118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:42.753861 6118 hybrid_clock.cc:648] HybridClock initialized: now 1790607102753846 us; error 33 us; skew 500 ppm
I20260928 14:51:42.755199 6118 webserver.cc:460] Webserver started at http://127.18.96.129:41289/ using document root <none> and password file <none>
I20260928 14:51:42.755450 6118 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:42.755520 6118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:42.755677 6118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:42.756893 6118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/data/instance:
uuid: "da4122c6dbcc4ca6847f8b9dc6fae474"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "6faea7329f42635276474813e21c16a4"
server_key_iv: "817f4b8b653de993882dff34258afb10"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.757259 6118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance:
uuid: "da4122c6dbcc4ca6847f8b9dc6fae474"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "6faea7329f42635276474813e21c16a4"
server_key_iv: "817f4b8b653de993882dff34258afb10"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.758698 6118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:51:42.759469 6132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.759696 6118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:42.759805 6118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/wal
uuid: "da4122c6dbcc4ca6847f8b9dc6fae474"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "6faea7329f42635276474813e21c16a4"
server_key_iv: "817f4b8b653de993882dff34258afb10"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.759917 6118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:42.767341 6118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:42.767647 6118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:42.767793 6118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:42.768112 6118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:42.768443 6118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:42.768502 6118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.768543 6118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:42.768570 6118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.774291 6118 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.129:33029
I20260928 14:51:42.774390 6245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.129:33029 every 8 connection(s)
I20260928 14:51:42.774713 6118 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/data/info.pb
I20260928 14:51:42.780305 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.775457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.129:35599 (local address 127.18.96.190:35689)
0928 14:51:42.775584 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.775586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.775798 (+ 212us) server_negotiation.cc:407] Connection header received
0928 14:51:42.776387 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.776388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.776402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.776447 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:42.777200 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.777708 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.778719 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.778916 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.779000 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.779975 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:42.779976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:42.779980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:42.779986 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:42.780000 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:42.780003 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:42.780068 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.780185 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.780220 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:51:42.780553 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.775434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35689 (local address 127.18.96.129:35599)
0928 14:51:42.775672 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:42.775707 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:42.776269 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:42.776523 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:42.776527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:42.776535 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:42.777085 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.777091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.777813 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.777815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.778625 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.778627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.779012 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.779013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.779148 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.779776 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:51:42.779776 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:42.779805 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:51:42.779828 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:42.780062 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:42.780070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:42.780074 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:42.780187 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:51:42.780297 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20260928 14:51:42.781169 6246 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35689
I20260928 14:51:42.781275 6246 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:42.781457 6246 heartbeater.cc:511] Master 127.18.96.190:35689 requested a full tablet report, sending...
I20260928 14:51:42.782040 6058 ts_manager.cc:194] Registered new tserver with Master: da4122c6dbcc4ca6847f8b9dc6fae474 (127.18.96.129:33029)
I20260928 14:51:42.782624 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 6118
I20260928 14:51:42.782732 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-0/wal/instance
I20260928 14:51:42.782943 18818 external_mini_cluster.cc:1428] Setting key 45848d18b56849785c6d6239c8363c8e
I20260928 14:51:42.782929 6058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.129:35599
I20260928 14:51:42.783995 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.130:0
--local_ip_for_outbound_sockets=127.18.96.130
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35689
--builtin_ntp_servers=127.18.96.148:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:51:42.867199 6249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:42.867419 6249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:42.867440 6249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:42.868674 6249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:42.868721 6249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:42.868746 6249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.130
I20260928 14:51:42.870127 6249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.130
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.130
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:42.870389 6249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:42.870715 6249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:42.875900 6255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:42.875900 6254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:42.876124 6249 server_base.cc:1034] running on GCE node
W20260928 14:51:42.876137 6257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:42.876356 6249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:42.876569 6249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:42.877714 6249 hybrid_clock.cc:648] HybridClock initialized: now 1790607102877699 us; error 29 us; skew 500 ppm
I20260928 14:51:42.879067 6249 webserver.cc:460] Webserver started at http://127.18.96.130:44691/ using document root <none> and password file <none>
I20260928 14:51:42.879323 6249 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:42.879369 6249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:42.879558 6249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:42.880702 6249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/data/instance:
uuid: "416dd2b957244f2b95f27b063f1cfa34"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "b7582cb5a596730f52bee6f78fa419a0"
server_key_iv: "9f326c648708b9bfa15443b87b2b9769"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.881052 6249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance:
uuid: "416dd2b957244f2b95f27b063f1cfa34"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "b7582cb5a596730f52bee6f78fa419a0"
server_key_iv: "9f326c648708b9bfa15443b87b2b9769"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.882402 6249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:51:42.883158 6263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.883323 6249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:51:42.883427 6249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/wal
uuid: "416dd2b957244f2b95f27b063f1cfa34"
format_stamp: "Formatted at 2026-09-28 14:51:42 on dist-test-slave-j968"
server_key: "b7582cb5a596730f52bee6f78fa419a0"
server_key_iv: "9f326c648708b9bfa15443b87b2b9769"
server_key_version: "encryptionkey@0"
I20260928 14:51:42.883527 6249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:42.899197 6249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:42.899556 6249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:42.899708 6249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:42.900033 6249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:42.900389 6249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:42.900440 6249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.900484 6249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:42.900518 6249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:42.906876 6249 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.130:40745
I20260928 14:51:42.906914 6376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.130:40745 every 8 connection(s)
I20260928 14:51:42.907327 6249 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/data/info.pb
I20260928 14:51:42.909214 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 6249
I20260928 14:51:42.909297 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/wal/instance
I20260928 14:51:42.909489 18818 external_mini_cluster.cc:1428] Setting key 9d72069f8fbc59257894ccdda58e338a
I20260928 14:51:42.910491 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.131:0
--local_ip_for_outbound_sockets=127.18.96.131
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35689
--builtin_ntp_servers=127.18.96.148:38957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:51:42.913136 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.908159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.130:39357 (local address 127.18.96.190:35689)
0928 14:51:42.908224 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:42.908226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:42.908425 (+ 199us) server_negotiation.cc:407] Connection header received
0928 14:51:42.908933 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:42.908934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:42.908946 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:42.909046 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:42.909852 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.910502 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.911579 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.911798 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.911874 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.912843 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:42.912845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:42.912845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:42.912850 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:42.912866 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:42.912871 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:42.912920 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:42.913014 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 14:51:42.913044 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:51:42.913420 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:42.908106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35689 (local address 127.18.96.130:39357)
0928 14:51:42.908298 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:42.908347 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:42.908835 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:42.909106 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:42.909109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:42.909118 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:42.909719 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.909725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.910622 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.910623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.911470 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:42.911473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:42.911924 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:42.911925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:42.912084 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:42.912682 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:51:42.912683 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:42.912724 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:51:42.912753 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:42.912934 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:42.912940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:42.912944 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:42.913013 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:51:42.913144 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260928 14:51:42.914057 6377 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35689
I20260928 14:51:42.914166 6377 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:42.914356 6377 heartbeater.cc:511] Master 127.18.96.190:35689 requested a full tablet report, sending...
I20260928 14:51:42.914768 6058 ts_manager.cc:194] Registered new tserver with Master: 416dd2b957244f2b95f27b063f1cfa34 (127.18.96.130:40745)
I20260928 14:51:42.915143 6058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.130:39357
W20260928 14:51:42.993521 6380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:42.993770 6380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:42.993795 6380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:42.995043 6380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:42.995092 6380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:42.995121 6380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.131
I20260928 14:51:42.996670 6380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:38957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.131
--webserver_port=0
--tserver_master_addrs=127.18.96.190:35689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.131
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:42.996937 6380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:42.997258 6380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:43.002694 6388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:43.002820 6380 server_base.cc:1034] running on GCE node
W20260928 14:51:43.002697 6386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:43.002697 6385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:43.003273 6380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:43.003561 6380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:43.004781 6380 hybrid_clock.cc:648] HybridClock initialized: now 1790607103004767 us; error 29 us; skew 500 ppm
I20260928 14:51:43.006395 6380 webserver.cc:460] Webserver started at http://127.18.96.131:40117/ using document root <none> and password file <none>
I20260928 14:51:43.006700 6380 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:43.006775 6380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:43.006932 6380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:43.008149 6380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/data/instance:
uuid: "f3b7d35c14e742bf93314c2375b21619"
format_stamp: "Formatted at 2026-09-28 14:51:43 on dist-test-slave-j968"
server_key: "e911d323a9f392be69257250481deb70"
server_key_iv: "e7ff7cb5025605356d16b7465890e00d"
server_key_version: "encryptionkey@0"
I20260928 14:51:43.008517 6380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance:
uuid: "f3b7d35c14e742bf93314c2375b21619"
format_stamp: "Formatted at 2026-09-28 14:51:43 on dist-test-slave-j968"
server_key: "e911d323a9f392be69257250481deb70"
server_key_iv: "e7ff7cb5025605356d16b7465890e00d"
server_key_version: "encryptionkey@0"
I20260928 14:51:43.009856 6380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:51:43.010762 6394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:43.011052 6380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:43.011261 6380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/wal
uuid: "f3b7d35c14e742bf93314c2375b21619"
format_stamp: "Formatted at 2026-09-28 14:51:43 on dist-test-slave-j968"
server_key: "e911d323a9f392be69257250481deb70"
server_key_iv: "e7ff7cb5025605356d16b7465890e00d"
server_key_version: "encryptionkey@0"
I20260928 14:51:43.011492 6380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:43.021530 6380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:43.021855 6380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:43.022012 6380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:43.022289 6380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:51:43.022655 6380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:51:43.022707 6380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:43.022743 6380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:51:43.022764 6380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:43.028944 6380 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.131:45535
I20260928 14:51:43.029072 6507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.131:45535 every 8 connection(s)
I20260928 14:51:43.029379 6380 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/data/info.pb
I20260928 14:51:43.034955 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.030211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:37065 (local address 127.18.96.190:35689)
0928 14:51:43.030287 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:43.030289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:43.030531 (+ 242us) server_negotiation.cc:407] Connection header received
0928 14:51:43.031099 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:43.031100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:43.031113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:43.031175 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:43.031888 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.032469 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.033552 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.033769 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.033853 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.034651 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:43.034652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:43.034653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:43.034658 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:43.034672 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:43.034677 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:43.034740 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:43.034836 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 14:51:43.034872 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:51:43.035156 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.030211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:35689 (local address 127.18.96.131:37065)
0928 14:51:43.030409 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:43.030443 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:43.030985 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:43.031252 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:43.031255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:43.031263 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:51:43.031777 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.031781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.032591 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.032592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.033466 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.033468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.033865 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.033866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.033995 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.034485 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:51:43.034485 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:51:43.034513 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:51:43.034536 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:51:43.034746 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:51:43.034752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:51:43.034756 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:51:43.034838 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:51:43.034942 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260928 14:51:43.035636 6508 heartbeater.cc:347] Connected to a master server at 127.18.96.190:35689
I20260928 14:51:43.035746 6508 heartbeater.cc:464] Registering TS with master...
I20260928 14:51:43.035998 6508 heartbeater.cc:511] Master 127.18.96.190:35689 requested a full tablet report, sending...
I20260928 14:51:43.036008 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 6380
I20260928 14:51:43.036175 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/wal/instance
I20260928 14:51:43.036415 18818 external_mini_cluster.cc:1428] Setting key c33bf90983d9b894430f587a6237c15a
I20260928 14:51:43.036408 6058 ts_manager.cc:194] Registered new tserver with Master: f3b7d35c14e742bf93314c2375b21619 (127.18.96.131:45535)
I20260928 14:51:43.036864 6058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.96.131:37065
I20260928 14:51:43.037359 18818 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:51:43.041597 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.038688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.18.96.190:35689)
0928 14:51:43.038770 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:43.038772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:43.038854 (+ 82us) server_negotiation.cc:407] Connection header received
0928 14:51:43.038888 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:43.038889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:43.038899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:43.038940 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:43.039075 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.039529 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.040371 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.040563 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.040630 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.041308 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:43.041310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:43.041310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:43.041314 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:43.041328 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:43.041332 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:43.041377 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:43.041412 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 14:51:43.041434 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:51:43.043128 6058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:51:43.044859 6058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:51:43.051591 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.047308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.18.96.130:40745)
0928 14:51:43.047838 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:43.047842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:43.047851 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:51:43.047925 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:43.047928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:43.047944 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:43.048176 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:43.048244 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.049387 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.050654 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.051171 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.051262 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.051364 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:43.051446 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 14:51:43.051488 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":97,"threads_started":1}
I20260928 14:51:43.051622 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.047289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.131:45535 (local address 127.0.0.1:53136)
0928 14:51:43.047493 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:43.047502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:43.047564 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:43.047998 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:43.047999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:43.048002 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:43.048096 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.048099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.049107 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.049108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.050378 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.050381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.051359 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.051360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.051429 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.051436 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:51:43.051468 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 14:51:43.051519 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":14,"thread_start_us":97,"threads_started":1}
I20260928 14:51:43.051813 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.047203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:40745 (local address 127.0.0.1:52240)
0928 14:51:43.047446 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:43.047493 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:43.047583 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:43.048036 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:43.048037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:43.048039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:43.048093 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.048095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.049497 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.049498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.050577 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.050579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.051262 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.051263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.051338 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.051352 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:51:43.051431 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:51:43.051498 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":52,"thread_start_us":88,"threads_started":1}
I20260928 14:51:43.052068 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.047539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.18.96.131:45535)
0928 14:51:43.047776 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:43.047779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:43.047789 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:51:43.047830 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:43.047831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:43.047844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:43.048184 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:43.048249 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.049015 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.050469 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.051253 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.051779 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.051913 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:43.051942 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 14:51:43.051987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260928 14:51:43.052080 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.047577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.18.96.129:33029)
0928 14:51:43.047750 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:43.047756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:43.047767 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:51:43.047813 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:43.047815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:43.047831 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:43.048176 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:43.048240 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.049411 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.050516 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.051248 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.051565 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.051833 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:43.051948 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 14:51:43.051987 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260928 14:51:43.052197 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.047291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:33029 (local address 127.0.0.1:51692)
0928 14:51:43.047670 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:43.047679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:43.047719 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:43.047992 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:43.047996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:43.047999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:43.048097 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.048098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.049508 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.049509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.050402 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.050405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.051357 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.051359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.051429 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.051437 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:51:43.051468 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 14:51:43.051519 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":21,"thread_start_us":101,"threads_started":1}
I20260928 14:51:43.052713 6311 tablet_service.cc:1467] Processing CreateTablet for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf (DEFAULT_TABLE table=test-table [id=b3aa29c7b90b4b3686e53936ef44f7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:43.052752 6442 tablet_service.cc:1467] Processing CreateTablet for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf (DEFAULT_TABLE table=test-table [id=b3aa29c7b90b4b3686e53936ef44f7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:43.053041 6442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:43.053043 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:43.055750 6522 tablet_bootstrap.cc:492] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: Bootstrap starting.
I20260928 14:51:43.055857 6523 tablet_bootstrap.cc:492] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: Bootstrap starting.
I20260928 14:51:43.056185 6180 tablet_service.cc:1467] Processing CreateTablet for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf (DEFAULT_TABLE table=test-table [id=b3aa29c7b90b4b3686e53936ef44f7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:51:43.056479 6522 tablet_bootstrap.cc:654] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:43.056500 6523 tablet_bootstrap.cc:654] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:43.056550 6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:43.056792 6523 log.cc:824] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:43.056811 6522 log.cc:824] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:43.057542 6523 tablet_bootstrap.cc:492] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: No bootstrap required, opened a new log
I20260928 14:51:43.057598 6522 tablet_bootstrap.cc:492] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: No bootstrap required, opened a new log
I20260928 14:51:43.057623 6523 ts_tablet_manager.cc:1397] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:51:43.057667 6522 ts_tablet_manager.cc:1397] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:51:43.059679 6523 raft_consensus.cc:348] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } }
I20260928 14:51:43.059840 6523 raft_consensus.cc:374] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:43.059911 6523 raft_consensus.cc:729] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b7d35c14e742bf93314c2375b21619, State: Initialized, Role: FOLLOWER
I20260928 14:51:43.060053 6523 consensus_queue.cc:260] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } }
I20260928 14:51:43.060329 6523 ts_tablet_manager.cc:1428] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:51:43.060453 6508 heartbeater.cc:503] Master 127.18.96.190:35689 was elected leader, sending a full tablet report...
I20260928 14:51:43.060425 6522 raft_consensus.cc:348] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } }
I20260928 14:51:43.060552 6526 tablet_bootstrap.cc:492] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474: Bootstrap starting.
I20260928 14:51:43.060583 6522 raft_consensus.cc:374] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:43.060621 6522 raft_consensus.cc:729] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416dd2b957244f2b95f27b063f1cfa34, State: Initialized, Role: FOLLOWER
I20260928 14:51:43.060755 6522 consensus_queue.cc:260] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } }
I20260928 14:51:43.060998 6522 ts_tablet_manager.cc:1428] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 14:51:43.061057 6377 heartbeater.cc:503] Master 127.18.96.190:35689 was elected leader, sending a full tablet report...
I20260928 14:51:43.061251 6526 tablet_bootstrap.cc:654] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:43.061605 6526 log.cc:824] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:43.062520 6526 tablet_bootstrap.cc:492] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474: No bootstrap required, opened a new log
I20260928 14:51:43.062639 6526 ts_tablet_manager.cc:1397] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:51:43.064692 6526 raft_consensus.cc:348] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } }
I20260928 14:51:43.064821 6526 raft_consensus.cc:374] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:43.064844 6526 raft_consensus.cc:729] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4122c6dbcc4ca6847f8b9dc6fae474, State: Initialized, Role: FOLLOWER
I20260928 14:51:43.064980 6526 consensus_queue.cc:260] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } }
I20260928 14:51:43.065241 6526 ts_tablet_manager.cc:1428] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:51:43.065297 6246 heartbeater.cc:503] Master 127.18.96.190:35689 was elected leader, sending a full tablet report...
W20260928 14:51:43.157893 6378 tablet.cc:2369] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:51:43.209703 6527 raft_consensus.cc:482] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:51:43.209889 6527 raft_consensus.cc:504] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } }
I20260928 14:51:43.210331 6527 leader_election.cc:288] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 416dd2b957244f2b95f27b063f1cfa34 (127.18.96.130:40745), da4122c6dbcc4ca6847f8b9dc6fae474 (127.18.96.129:33029)
I20260928 14:51:43.213455 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.210683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:40263 (local address 127.18.96.130:40745)
0928 14:51:43.210726 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:43.210728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:43.210736 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:51:43.210770 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:43.210771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:43.210779 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:43.210849 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:43.211026 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.211543 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.212686 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.213063 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.213130 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.213161 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:43.213328 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:51:43.213372 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:51:43.213474 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.210660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.96.131:45705 (local address 127.18.96.129:33029)
0928 14:51:43.210723 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:43.210726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:43.210923 (+ 197us) server_negotiation.cc:407] Connection header received
0928 14:51:43.211002 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:43.211003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:43.211015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:43.211124 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:51:43.211288 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.211805 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.212758 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.213127 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.213206 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.213213 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:43.213358 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 14:51:43.213398 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:51:43.213491 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.210555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:40745 (local address 127.18.96.131:40263)
0928 14:51:43.210622 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:43.210634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:43.210681 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:43.210854 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:43.210854 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:43.210856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:43.210938 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.210940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.211633 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.211634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.212566 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.212570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.213189 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.213190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.213250 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.213253 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:51:43.213322 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:51:43.213367 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 14:51:43.213510 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.210665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:33029 (local address 127.18.96.131:45705)
0928 14:51:43.210844 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:43.210857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:51:43.210933 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:51:43.211128 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:51:43.211129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:51:43.211131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:51:43.211207 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.211209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.211913 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.211914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.212665 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:51:43.212667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.213206 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.213207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:51:43.213286 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.213289 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:51:43.213352 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:51:43.213392 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":130,"threads_started":1}
I20260928 14:51:43.213877 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f20e214e55d47cd9ed7a2f1c7a3c3cf" candidate_uuid: "f3b7d35c14e742bf93314c2375b21619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416dd2b957244f2b95f27b063f1cfa34" is_pre_election: true
I20260928 14:51:43.214064 6331 raft_consensus.cc:2454] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b7d35c14e742bf93314c2375b21619 in term 0.
I20260928 14:51:43.214093 6200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f20e214e55d47cd9ed7a2f1c7a3c3cf" candidate_uuid: "f3b7d35c14e742bf93314c2375b21619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" is_pre_election: true
I20260928 14:51:43.214239 6200 raft_consensus.cc:2454] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b7d35c14e742bf93314c2375b21619 in term 0.
I20260928 14:51:43.214315 6397 leader_election.cc:302] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416dd2b957244f2b95f27b063f1cfa34, f3b7d35c14e742bf93314c2375b21619; no voters:
I20260928 14:51:43.214457 6527 raft_consensus.cc:2781] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:51:43.214519 6527 raft_consensus.cc:482] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:51:43.214537 6527 raft_consensus.cc:3037] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:43.215329 6527 raft_consensus.cc:504] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } }
I20260928 14:51:43.215485 6527 leader_election.cc:288] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [CANDIDATE]: Term 1 election: Requested vote from peers 416dd2b957244f2b95f27b063f1cfa34 (127.18.96.130:40745), da4122c6dbcc4ca6847f8b9dc6fae474 (127.18.96.129:33029)
I20260928 14:51:43.215679 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f20e214e55d47cd9ed7a2f1c7a3c3cf" candidate_uuid: "f3b7d35c14e742bf93314c2375b21619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "416dd2b957244f2b95f27b063f1cfa34"
I20260928 14:51:43.215747 6200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f20e214e55d47cd9ed7a2f1c7a3c3cf" candidate_uuid: "f3b7d35c14e742bf93314c2375b21619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474"
I20260928 14:51:43.215801 6331 raft_consensus.cc:3037] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:43.215811 6200 raft_consensus.cc:3037] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:43.216747 6200 raft_consensus.cc:2454] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b7d35c14e742bf93314c2375b21619 in term 1.
I20260928 14:51:43.216754 6331 raft_consensus.cc:2454] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b7d35c14e742bf93314c2375b21619 in term 1.
I20260928 14:51:43.216933 6395 leader_election.cc:302] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da4122c6dbcc4ca6847f8b9dc6fae474, f3b7d35c14e742bf93314c2375b21619; no voters:
I20260928 14:51:43.217027 6527 raft_consensus.cc:2781] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:43.217183 6527 raft_consensus.cc:686] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 1 LEADER]: Becoming Leader. State: Replica: f3b7d35c14e742bf93314c2375b21619, State: Running, Role: LEADER
I20260928 14:51:43.217301 6527 consensus_queue.cc:237] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } }
I20260928 14:51:43.218202 6058 catalog_manager.cc:5495] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3b7d35c14e742bf93314c2375b21619 (127.18.96.131). New cstate: current_term: 1 leader_uuid: "f3b7d35c14e742bf93314c2375b21619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3b7d35c14e742bf93314c2375b21619" member_type: VOTER last_known_addr { host: "127.18.96.131" port: 45535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 } health_report { overall_health: UNKNOWN } } }
I20260928 14:51:43.233129 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.230345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.18.96.131:45535)
0928 14:51:43.230410 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:43.230411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:43.230420 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:51:43.230490 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:43.230491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:43.230511 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:43.230589 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:51:43.230749 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.231328 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.232202 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:43.232411 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:43.232475 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:43.232669 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:51:43.232789 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:51:43.232858 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:43.232945 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 14:51:43.232967 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:51:43.235152 6331 raft_consensus.cc:1261] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [term 1 FOLLOWER]: Refusing update from remote peer f3b7d35c14e742bf93314c2375b21619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:43.235176 6200 raft_consensus.cc:1261] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P da4122c6dbcc4ca6847f8b9dc6fae474 [term 1 FOLLOWER]: Refusing update from remote peer f3b7d35c14e742bf93314c2375b21619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:51:43.235558 6527 consensus_queue.cc:1035] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da4122c6dbcc4ca6847f8b9dc6fae474" member_type: VOTER last_known_addr { host: "127.18.96.129" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:43.235719 6532 consensus_queue.cc:1035] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "416dd2b957244f2b95f27b063f1cfa34" member_type: VOTER last_known_addr { host: "127.18.96.130" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:51:43.238494 6536 mvcc.cc:205] Tried to move back new op lower bound from 7334326694848212992 to 7334326694778826752. Current Snapshot: MvccSnapshot[applied={T|T < 7334326694848212992}]
I20260928 14:51:43.238555 6537 mvcc.cc:205] Tried to move back new op lower bound from 7334326694848212992 to 7334326694778826752. Current Snapshot: MvccSnapshot[applied={T|T < 7334326694848212992}]
I20260928 14:51:43.240710 6057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:43.240794 6057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:51:43.241627 6057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b7b48bee35d4185b23c97ff2d632630: Sending DeleteTablet for 3 replicas of tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf
I20260928 14:51:43.241932 6442 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:43 UTC) from {username='slave'} at 127.0.0.1:53136
I20260928 14:51:43.241989 6311 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:43 UTC) from {username='slave'} at 127.0.0.1:52240
I20260928 14:51:43.242229 6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:51:43 UTC) from {username='slave'} at 127.0.0.1:51692
I20260928 14:51:43.242386 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 6118
I20260928 14:51:43.248030 6545 tablet_replica.cc:330] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: stopping tablet replica
I20260928 14:51:43.248188 6545 raft_consensus.cc:2229] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:51:43.248286 6545 raft_consensus.cc:2258] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:43.248466 6546 tablet_replica.cc:330] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: stopping tablet replica
I20260928 14:51:43.248739 6546 raft_consensus.cc:2229] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:51:43.248817 6545 ts_tablet_manager.cc:1905] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:51:43.248862 6546 raft_consensus.cc:2258] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:51:43.249372 6546 ts_tablet_manager.cc:1905] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:51:43.250222 6042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.129:33029 (error 108)
I20260928 14:51:43.250872 6545 ts_tablet_manager.cc:1918] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:43.250950 6545 log.cc:1196] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-2/wal/wals/7f20e214e55d47cd9ed7a2f1c7a3c3cf
I20260928 14:51:43.251077 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.250898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.129:33029 (local address 127.0.0.1:51702)
0928 14:51:43.250994 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:43.251040 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.129:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 14:51:43.251111 6546 ts_tablet_manager.cc:1918] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:51:43.251180 6546 log.cc:1196] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: Deleting WAL directory at /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606857360589-18818-0/minicluster-data/ts-1/wal/wals/7f20e214e55d47cd9ed7a2f1c7a3c3cf
I20260928 14:51:43.251228 6545 ts_tablet_manager.cc:1939] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P f3b7d35c14e742bf93314c2375b21619: Deleting consensus metadata
I20260928 14:51:43.251416 6546 ts_tablet_manager.cc:1939] T 7f20e214e55d47cd9ed7a2f1c7a3c3cf P 416dd2b957244f2b95f27b063f1cfa34: Deleting consensus metadata
W20260928 14:51:43.251401 6042 catalog_manager.cc:4584] TS da4122c6dbcc4ca6847f8b9dc6fae474 (127.18.96.129:33029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf: Network error: Client connection negotiation failed: client connection to 127.18.96.129:33029: connect: Connection refused (error 111)
I20260928 14:51:43.251643 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 6249
I20260928 14:51:43.251667 6043 catalog_manager.cc:4841] TS f3b7d35c14e742bf93314c2375b21619 (127.18.96.131:45535): tablet 7f20e214e55d47cd9ed7a2f1c7a3c3cf (table test-table [id=b3aa29c7b90b4b3686e53936ef44f7c7]) successfully deleted
I20260928 14:51:43.256985 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 6380
I20260928 14:51:43.257128 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:43.256968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.130:40745 (local address 127.0.0.1:52252)
0928 14:51:43.257041 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 14:51:43.257082 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.130:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 14:51:43.262621 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 6027
2026-09-28T14:51:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (799 ms)
[----------] 6 tests from EncryptionPolicyTest (8717 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:43 dist-test-slave-j968 krb5kdc[6551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:43 dist-test-slave-j968 krb5kdc[6551](info): set up 2 sockets
Sep 28 14:51:43 dist-test-slave-j968 krb5kdc[6551](info): commencing operation
krb5kdc: starting...
W20260928 14:51:44.298806 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:44 dist-test-slave-j968 krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:44.427402 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:44.513029 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:44.513314 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:44.513356 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:44.514756 6564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:44.514829 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:44.514854 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:44.514885 6564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:44.514927 6564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:44.516626 6564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:44.516934 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:44.517189 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:44.522744 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:44.522739 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:44.522778 6564 server_base.cc:1034] running on GCE node
W20260928 14:51:44.522744 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:44.523378 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:44.523782 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:44.524937 6564 hybrid_clock.cc:648] HybridClock initialized: now 1790607104524924 us; error 74 us; skew 500 ppm
Sep 28 14:51:44 dist-test-slave-j968 krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:44.527832 6564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:44.528469 6564 webserver.cc:460] Webserver started at http://127.0.0.1:35303/ using document root <none> and password file <none>
I20260928 14:51:44.528843 6564 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:44.528951 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:44.529120 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:44.530416 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "75bddd7343514cb5883e97e08d800f94"
format_stamp: "Formatted at 2026-09-28 14:51:44 on dist-test-slave-j968"
server_key: "7d159170d1fe37025a0598745a3e7679"
server_key_iv: "99fa2b04df7bc87cbb1de594aa287dbd"
server_key_version: "encryptionkey@0"
I20260928 14:51:44.530859 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "75bddd7343514cb5883e97e08d800f94"
format_stamp: "Formatted at 2026-09-28 14:51:44 on dist-test-slave-j968"
server_key: "7d159170d1fe37025a0598745a3e7679"
server_key_iv: "99fa2b04df7bc87cbb1de594aa287dbd"
server_key_version: "encryptionkey@0"
I20260928 14:51:44.532433 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 14:51:44.533304 6579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:44.533500 6564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:44.533612 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "75bddd7343514cb5883e97e08d800f94"
format_stamp: "Formatted at 2026-09-28 14:51:44 on dist-test-slave-j968"
server_key: "7d159170d1fe37025a0598745a3e7679"
server_key_iv: "99fa2b04df7bc87cbb1de594aa287dbd"
server_key_version: "encryptionkey@0"
I20260928 14:51:44.533691 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:44.555153 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:44.557585 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:44.557772 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:44.562448 6564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33145
I20260928 14:51:44.562482 6631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33145 every 8 connection(s)
I20260928 14:51:44.562923 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:44.563898 6632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:44.566684 6632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94: Bootstrap starting.
I20260928 14:51:44.567306 6632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:44.567605 6632 log.cc:824] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:44.568423 6632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94: No bootstrap required, opened a new log
I20260928 14:51:44.570008 6632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bddd7343514cb5883e97e08d800f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33145 } }
I20260928 14:51:44.570184 6632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:44.570231 6632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bddd7343514cb5883e97e08d800f94, State: Initialized, Role: FOLLOWER
I20260928 14:51:44.570396 6632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bddd7343514cb5883e97e08d800f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33145 } }
I20260928 14:51:44.570478 6632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:44.570523 6632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:44.570575 6632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:44.571162 6632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bddd7343514cb5883e97e08d800f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33145 } }
I20260928 14:51:44.571288 6632 leader_election.cc:302] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75bddd7343514cb5883e97e08d800f94; no voters:
I20260928 14:51:44.571467 6632 leader_election.cc:288] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:44.571560 6635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:44.571733 6632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:44.571754 6635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [term 1 LEADER]: Becoming Leader. State: Replica: 75bddd7343514cb5883e97e08d800f94, State: Running, Role: LEADER
I20260928 14:51:44.571857 6635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bddd7343514cb5883e97e08d800f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33145 } }
I20260928 14:51:44.572332 6637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75bddd7343514cb5883e97e08d800f94. Latest consensus state: current_term: 1 leader_uuid: "75bddd7343514cb5883e97e08d800f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bddd7343514cb5883e97e08d800f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33145 } } }
I20260928 14:51:44.572420 6637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:44.572482 6636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75bddd7343514cb5883e97e08d800f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bddd7343514cb5883e97e08d800f94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33145 } } }
I20260928 14:51:44.572552 6636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:44.572686 6644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:44.572922 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 6564
I20260928 14:51:44.573032 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:44.573237 18818 external_mini_cluster.cc:1428] Setting key 573fbb5afbd41d28702fb25e70145c53
I20260928 14:51:44.573421 6644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:44.575166 6644 catalog_manager.cc:1347] Generated new cluster ID: 1c2b26a83afa4dddad693be2508d91f6
I20260928 14:51:44.575227 6644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:51:44 dist-test-slave-j968 krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:51:44.585464 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:44.573752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39465 (local address 127.0.0.1:33145)
0928 14:51:44.574167 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:44.574175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:44.574200 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:51:44.574289 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:44.574295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:44.574339 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:44.574516 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:44.575313 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:44.576744 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:44.577797 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:44.578419 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:44.578505 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:44.581469 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:44.581474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:44.581474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:44.581488 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:44.583492 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:44.583990 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:44.583993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:44.583993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:44.584097 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:44.584308 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:44.584309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:44.584309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:44.584602 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:44.584730 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:44.585128 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 14:51:44.585226 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":71,"threads_started":1}
I20260928 14:51:44.586369 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:44.595669 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:44.588108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60275 (local address 127.0.0.1:33145)
0928 14:51:44.588149 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:44.588151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:44.588412 (+ 261us) server_negotiation.cc:407] Connection header received
0928 14:51:44.588473 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:44.588474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:44.588484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:44.588536 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:44.589190 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:44.589937 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:44.591004 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:44.591312 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:44.591376 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:44.592885 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:44.592886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:44.592887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:44.592894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:44.594443 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:44.594878 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:44.594879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:44.594879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:44.594933 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:44.595162 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:44.595163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:44.595163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:44.595280 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:44.595365 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:44.595517 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 14:51:44.595577 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:51:44.609352 6644 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:44.609935 6644 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:44.615374 6644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75bddd7343514cb5883e97e08d800f94: Generated new TSK 0
I20260928 14:51:44.615618 6644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:44.625615 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 6564
W20260928 14:51:44.630260 6549 connection.cc:535] client connection to 127.0.0.1:33145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33145): Connection reset by peer (error 104)
2026-09-28T14:51:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:44 dist-test-slave-j968 krb5kdc[6663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:44 dist-test-slave-j968 krb5kdc[6663](info): set up 2 sockets
Sep 28 14:51:44 dist-test-slave-j968 krb5kdc[6663](info): commencing operation
krb5kdc: starting...
W20260928 14:51:45.701526 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:45 dist-test-slave-j968 krb5kdc[6663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:45.827169 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:45.910465 6676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:45.910684 6676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:45.910708 6676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:45.911819 6676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:45.911859 6676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:45.911943 6676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:45.911959 6676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:45.911978 6676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:45.913533 6676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42545
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:45.913738 6676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:45.913928 6676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:45.919191 6684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:45.919261 6676 server_base.cc:1034] running on GCE node
W20260928 14:51:45.919184 6681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:45.919184 6682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:45.919724 6676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:45.920073 6676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:45.921247 6676 hybrid_clock.cc:648] HybridClock initialized: now 1790607105921232 us; error 32 us; skew 500 ppm
Sep 28 14:51:45 dist-test-slave-j968 krb5kdc[6663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:45.923936 6676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:45.924414 6676 webserver.cc:460] Webserver started at http://127.0.0.1:39011/ using document root <none> and password file <none>
I20260928 14:51:45.924606 6676 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:45.924649 6676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:45.924738 6676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:45.925777 6676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "ac4ebbf694b74916acb49aabad6730f5"
format_stamp: "Formatted at 2026-09-28 14:51:45 on dist-test-slave-j968"
server_key: "95121eba840a5ddd88c5a8e3e2a8da6f"
server_key_iv: "d93369093a144c402a7687c82fcd9553"
server_key_version: "encryptionkey@0"
I20260928 14:51:45.926091 6676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "ac4ebbf694b74916acb49aabad6730f5"
format_stamp: "Formatted at 2026-09-28 14:51:45 on dist-test-slave-j968"
server_key: "95121eba840a5ddd88c5a8e3e2a8da6f"
server_key_iv: "d93369093a144c402a7687c82fcd9553"
server_key_version: "encryptionkey@0"
I20260928 14:51:45.927386 6676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:51:45.928123 6691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:45.928328 6676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:45.928450 6676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "ac4ebbf694b74916acb49aabad6730f5"
format_stamp: "Formatted at 2026-09-28 14:51:45 on dist-test-slave-j968"
server_key: "95121eba840a5ddd88c5a8e3e2a8da6f"
server_key_iv: "d93369093a144c402a7687c82fcd9553"
server_key_version: "encryptionkey@0"
I20260928 14:51:45.928545 6676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:45.941092 6676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:45.943421 6676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:45.943607 6676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:45.948288 6676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42545
I20260928 14:51:45.948318 6743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42545 every 8 connection(s)
I20260928 14:51:45.948714 6676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:45.949628 6744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:45.952431 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 6676
I20260928 14:51:45.952551 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:45.952669 6744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5: Bootstrap starting.
I20260928 14:51:45.952775 18818 external_mini_cluster.cc:1428] Setting key bf383490ae2077f7a2ef82c9c882f045
I20260928 14:51:45.953418 6744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:45.953730 6744 log.cc:824] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:45.954454 6744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5: No bootstrap required, opened a new log
I20260928 14:51:45.957059 6744 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ebbf694b74916acb49aabad6730f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42545 } }
I20260928 14:51:45.957252 6744 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:45.957288 6744 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4ebbf694b74916acb49aabad6730f5, State: Initialized, Role: FOLLOWER
I20260928 14:51:45.957424 6744 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ebbf694b74916acb49aabad6730f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42545 } }
I20260928 14:51:45.957504 6744 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:45.957538 6744 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:45.957597 6744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:45.958441 6744 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ebbf694b74916acb49aabad6730f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42545 } }
I20260928 14:51:45.958580 6744 leader_election.cc:302] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4ebbf694b74916acb49aabad6730f5; no voters:
I20260928 14:51:45.958766 6744 leader_election.cc:288] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:45.958798 6749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:45.958997 6749 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [term 1 LEADER]: Becoming Leader. State: Replica: ac4ebbf694b74916acb49aabad6730f5, State: Running, Role: LEADER
I20260928 14:51:45.959060 6744 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:45.959149 6749 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ebbf694b74916acb49aabad6730f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42545 } }
I20260928 14:51:45.959484 6749 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4ebbf694b74916acb49aabad6730f5. Latest consensus state: current_term: 1 leader_uuid: "ac4ebbf694b74916acb49aabad6730f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ebbf694b74916acb49aabad6730f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42545 } } }
I20260928 14:51:45.959600 6749 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:45.959497 6751 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac4ebbf694b74916acb49aabad6730f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4ebbf694b74916acb49aabad6730f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42545 } } }
I20260928 14:51:45.959693 6751 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:45.959901 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:45.953316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45759 (local address 127.0.0.1:42545)
0928 14:51:45.953565 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:45.953576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:45.953612 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:51:45.954173 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:45.954181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:45.954233 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:45.954385 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:45.954617 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:45.956212 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:45.957015 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:45.957709 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:45.957834 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:45.958644 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:45.958649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:45.958649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:45.958889 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:45.958933 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:45.958950 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:45.959040 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:45.959340 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 14:51:45.959372 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
I20260928 14:51:45.960395 6755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:45.960897 6755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:45.961094 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:45.962333 6755 catalog_manager.cc:1347] Generated new cluster ID: 3caca7445add44f99c37c849ee092e12
I20260928 14:51:45.962376 6755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:45.966637 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:45.962758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45355 (local address 127.0.0.1:42545)
0928 14:51:45.962798 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:45.962800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:45.962954 (+ 154us) server_negotiation.cc:407] Connection header received
0928 14:51:45.963021 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:45.963022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:45.963031 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:45.963084 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:45.963241 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:45.964009 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:45.965199 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:45.965519 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:45.965583 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:45.966359 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:45.966361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:45.966361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:45.966366 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:45.966381 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:45.966385 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:45.966453 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:45.966552 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 14:51:45.966563 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:51:45.987788 6755 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:45.988670 6755 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:45.993439 6755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac4ebbf694b74916acb49aabad6730f5: Generated new TSK 0
I20260928 14:51:45.993652 6755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:45.998921 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 6676
W20260928 14:51:46.003492 6661 connection.cc:535] client connection to 127.0.0.1:42545 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42545): Connection reset by peer (error 104)
2026-09-28T14:51:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:46 dist-test-slave-j968 krb5kdc[6775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:46 dist-test-slave-j968 krb5kdc[6775](info): set up 2 sockets
Sep 28 14:51:46 dist-test-slave-j968 krb5kdc[6775](info): commencing operation
krb5kdc: starting...
W20260928 14:51:47.040939 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:47 dist-test-slave-j968 krb5kdc[6775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:47.166453 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:47.256158 6788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:47.256366 6788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:47.256389 6788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:47.257496 6788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:47.257532 6788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:47.257546 6788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:47.257560 6788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:47.257571 6788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:47.259028 6788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:47.259224 6788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:47.259399 6788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:47.264469 6794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:47.264487 6796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:47.264599 6788 server_base.cc:1034] running on GCE node
W20260928 14:51:47.264485 6793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:47.265019 6788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:47.265334 6788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:47.266507 6788 hybrid_clock.cc:648] HybridClock initialized: now 1790607107266475 us; error 48 us; skew 500 ppm
Sep 28 14:51:47 dist-test-slave-j968 krb5kdc[6775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:47.269529 6788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:47.270032 6788 webserver.cc:460] Webserver started at http://127.0.0.1:33703/ using document root <none> and password file <none>
I20260928 14:51:47.270272 6788 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:47.270341 6788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:47.270491 6788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:47.271618 6788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "fa9d2e61f8f844d8aec744528f6c5ebc"
format_stamp: "Formatted at 2026-09-28 14:51:47 on dist-test-slave-j968"
server_key: "d168944b563d739de67bb7cb6a337dfb"
server_key_iv: "b7825abc604a0c4c82652025c643d62e"
server_key_version: "encryptionkey@0"
I20260928 14:51:47.272028 6788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "fa9d2e61f8f844d8aec744528f6c5ebc"
format_stamp: "Formatted at 2026-09-28 14:51:47 on dist-test-slave-j968"
server_key: "d168944b563d739de67bb7cb6a337dfb"
server_key_iv: "b7825abc604a0c4c82652025c643d62e"
server_key_version: "encryptionkey@0"
I20260928 14:51:47.273515 6788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:47.274330 6803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:47.274639 6788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:47.274765 6788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "fa9d2e61f8f844d8aec744528f6c5ebc"
format_stamp: "Formatted at 2026-09-28 14:51:47 on dist-test-slave-j968"
server_key: "d168944b563d739de67bb7cb6a337dfb"
server_key_iv: "b7825abc604a0c4c82652025c643d62e"
server_key_version: "encryptionkey@0"
I20260928 14:51:47.274847 6788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:47.278534 6788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:47.281105 6788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:47.281275 6788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:47.286119 6788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37063
I20260928 14:51:47.286154 6855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37063 every 8 connection(s)
I20260928 14:51:47.286520 6788 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:47.287448 6856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:47.290023 6856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc: Bootstrap starting.
I20260928 14:51:47.290613 6856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:47.290864 6856 log.cc:824] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:47.291545 6856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc: No bootstrap required, opened a new log
I20260928 14:51:47.292026 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 6788
I20260928 14:51:47.292229 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:47.292459 18818 external_mini_cluster.cc:1428] Setting key fb42be617c1759b7cc519de1401957d1
I20260928 14:51:47.294240 6856 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9d2e61f8f844d8aec744528f6c5ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37063 } }
I20260928 14:51:47.294431 6856 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:47.294472 6856 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9d2e61f8f844d8aec744528f6c5ebc, State: Initialized, Role: FOLLOWER
I20260928 14:51:47.294646 6856 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9d2e61f8f844d8aec744528f6c5ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37063 } }
I20260928 14:51:47.294764 6856 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:47.294807 6856 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:47.294898 6856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:47.295652 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:47.293001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56469 (local address 127.0.0.1:37063)
0928 14:51:47.293171 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:47.293179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:47.293264 (+ 85us) server_negotiation.cc:407] Connection header received
0928 14:51:47.294216 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:47.294223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:47.294252 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:47.294379 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:47.295122 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:47.295125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:47.295126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:47.295130 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:47.295176 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:47.295190 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:47.295247 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:47.295361 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 14:51:47.295422 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260928 14:51:47.296092 6856 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9d2e61f8f844d8aec744528f6c5ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37063 } }
I20260928 14:51:47.296202 6856 leader_election.cc:302] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa9d2e61f8f844d8aec744528f6c5ebc; no voters:
I20260928 14:51:47.296351 6856 leader_election.cc:288] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:47.296413 6861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:47.296597 6861 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [term 1 LEADER]: Becoming Leader. State: Replica: fa9d2e61f8f844d8aec744528f6c5ebc, State: Running, Role: LEADER
I20260928 14:51:47.296651 6856 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:47.296741 6861 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9d2e61f8f844d8aec744528f6c5ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37063 } }
I20260928 14:51:47.296808 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:47.297297 6862 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa9d2e61f8f844d8aec744528f6c5ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9d2e61f8f844d8aec744528f6c5ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37063 } } }
I20260928 14:51:47.297381 6862 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:47.297544 6863 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa9d2e61f8f844d8aec744528f6c5ebc. Latest consensus state: current_term: 1 leader_uuid: "fa9d2e61f8f844d8aec744528f6c5ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa9d2e61f8f844d8aec744528f6c5ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37063 } } }
I20260928 14:51:47.297610 6863 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:47.298023 6870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:47.298542 6870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:47.300305 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:47.298689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50587 (local address 127.0.0.1:37063)
0928 14:51:47.298729 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:47.298731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:47.298985 (+ 254us) server_negotiation.cc:407] Connection header received
0928 14:51:47.299109 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:47.299110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:47.299135 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:47.299176 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:47.300051 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:47.300052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:47.300053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:47.300058 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:47.300074 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:47.300078 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:47.300130 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:47.300227 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 14:51:47.300244 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:51:47.300800 6870 catalog_manager.cc:1347] Generated new cluster ID: e1ee66308781488d98e8774f45eba56a
I20260928 14:51:47.300858 6870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:47.314508 6870 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:47.314733 6870 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:47.320058 6870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa9d2e61f8f844d8aec744528f6c5ebc: Generated new TSK 0
I20260928 14:51:47.320320 6870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:47.326145 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 6788
2026-09-28T14:51:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:47 dist-test-slave-j968 krb5kdc[6887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:47 dist-test-slave-j968 krb5kdc[6887](info): set up 2 sockets
Sep 28 14:51:47 dist-test-slave-j968 krb5kdc[6887](info): commencing operation
krb5kdc: starting...
W20260928 14:51:48.401037 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:48 dist-test-slave-j968 krb5kdc[6887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:48.527629 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:48.610603 6900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:48.610832 6900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:48.610857 6900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:48.612119 6900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:48.612167 6900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:48.612186 6900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:48.612200 6900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:48.612211 6900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:48.613790 6900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:48.614068 6900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:48.614305 6900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:48.619534 6908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:48.619565 6906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:48.619653 6905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:48.620128 6900 server_base.cc:1034] running on GCE node
I20260928 14:51:48.620443 6900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:48.620811 6900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:48.622013 6900 hybrid_clock.cc:648] HybridClock initialized: now 1790607108622003 us; error 61 us; skew 500 ppm
Sep 28 14:51:48 dist-test-slave-j968 krb5kdc[6887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:48.630317 6900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:48.630905 6900 webserver.cc:460] Webserver started at http://127.0.0.1:39899/ using document root <none> and password file <none>
I20260928 14:51:48.631171 6900 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:48.631245 6900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:48.631407 6900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:48.632712 6900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "998ec8efa4e1428abaef4c6bb1124f0f"
format_stamp: "Formatted at 2026-09-28 14:51:48 on dist-test-slave-j968"
server_key: "f6e947b9a9d13be52d22eba1b52cbf83"
server_key_iv: "fd341e042391a7050723656e4f8c3fbf"
server_key_version: "encryptionkey@0"
I20260928 14:51:48.633123 6900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "998ec8efa4e1428abaef4c6bb1124f0f"
format_stamp: "Formatted at 2026-09-28 14:51:48 on dist-test-slave-j968"
server_key: "f6e947b9a9d13be52d22eba1b52cbf83"
server_key_iv: "fd341e042391a7050723656e4f8c3fbf"
server_key_version: "encryptionkey@0"
I20260928 14:51:48.634610 6900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:51:48.635489 6915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:48.635712 6900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:48.635814 6900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "998ec8efa4e1428abaef4c6bb1124f0f"
format_stamp: "Formatted at 2026-09-28 14:51:48 on dist-test-slave-j968"
server_key: "f6e947b9a9d13be52d22eba1b52cbf83"
server_key_iv: "fd341e042391a7050723656e4f8c3fbf"
server_key_version: "encryptionkey@0"
I20260928 14:51:48.635921 6900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:48.648531 6900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:48.651126 6900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:48.651253 6900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:48.655656 6900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46125
I20260928 14:51:48.655690 6967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46125 every 8 connection(s)
I20260928 14:51:48.656119 6900 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:48.657099 6968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:48.659451 6968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f: Bootstrap starting.
I20260928 14:51:48.660102 6968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:48.660385 6968 log.cc:824] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:48.661163 6968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f: No bootstrap required, opened a new log
I20260928 14:51:48.662798 6968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998ec8efa4e1428abaef4c6bb1124f0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46125 } }
I20260928 14:51:48.662961 6968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:48.663044 6968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998ec8efa4e1428abaef4c6bb1124f0f, State: Initialized, Role: FOLLOWER
I20260928 14:51:48.663228 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 6900
I20260928 14:51:48.663214 6968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998ec8efa4e1428abaef4c6bb1124f0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46125 } }
I20260928 14:51:48.663300 6968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:48.663332 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:48.663336 6968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:48.663381 6968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:48.663547 18818 external_mini_cluster.cc:1428] Setting key dcc36d9383fb11cf0708c18b9f0695a9
I20260928 14:51:48.664155 6968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998ec8efa4e1428abaef4c6bb1124f0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46125 } }
I20260928 14:51:48.664287 6968 leader_election.cc:302] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 998ec8efa4e1428abaef4c6bb1124f0f; no voters:
I20260928 14:51:48.664564 6968 leader_election.cc:288] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:48.664590 6973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:48.664693 6973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [term 1 LEADER]: Becoming Leader. State: Replica: 998ec8efa4e1428abaef4c6bb1124f0f, State: Running, Role: LEADER
I20260928 14:51:48.664837 6968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:48.664850 6973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998ec8efa4e1428abaef4c6bb1124f0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46125 } }
I20260928 14:51:48.665441 6974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "998ec8efa4e1428abaef4c6bb1124f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998ec8efa4e1428abaef4c6bb1124f0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46125 } } }
I20260928 14:51:48.665551 6974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:48.666261 6982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:48.666247 6975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 998ec8efa4e1428abaef4c6bb1124f0f. Latest consensus state: current_term: 1 leader_uuid: "998ec8efa4e1428abaef4c6bb1124f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998ec8efa4e1428abaef4c6bb1124f0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46125 } } }
I20260928 14:51:48.666529 6975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:48.666893 6982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:48.669416 6982 catalog_manager.cc:1347] Generated new cluster ID: a35f62b91f564ba3a86d959d3288350e
I20260928 14:51:48.669485 6982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:51:48 dist-test-slave-j968 krb5kdc[6887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:51:48.676011 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:48.664225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51827 (local address 127.0.0.1:46125)
0928 14:51:48.664419 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:48.664430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:48.664470 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:51:48.664632 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:48.664640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:48.664721 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:48.665785 (+ 1064us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:48.665992 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:48.667680 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:48.668928 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:48.669598 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:48.669744 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:48.672231 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:48.672241 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:48.672242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:48.672253 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:48.674357 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:48.674746 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:48.674750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:48.674750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:48.674861 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:48.675081 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:48.675082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:48.675083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:48.675347 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:48.675547 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:48.675618 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 14:51:48.675764 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260928 14:51:48.676995 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:48.691556 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:48.678388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44441 (local address 127.0.0.1:46125)
0928 14:51:48.678429 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:48.678430 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:48.684158 (+ 5728us) server_negotiation.cc:407] Connection header received
0928 14:51:48.684251 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:48.684252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:48.684265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:48.684323 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:48.684948 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:48.685527 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:48.686776 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:48.687025 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:48.687112 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:48.688463 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:48.688465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:48.688465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:48.688473 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:48.690169 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:48.690619 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:48.690620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:48.690621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:48.690690 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:48.690924 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:48.690925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:48.690925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:48.691064 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:48.691201 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:48.691323 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:51:48.691418 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:51:48.700850 6982 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:48.701599 6982 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:48.705493 6982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 998ec8efa4e1428abaef4c6bb1124f0f: Generated new TSK 0
I20260928 14:51:48.705730 6982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:48.714068 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 6900
2026-09-28T14:51:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:48 dist-test-slave-j968 krb5kdc[6999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:48 dist-test-slave-j968 krb5kdc[6999](info): set up 2 sockets
Sep 28 14:51:48 dist-test-slave-j968 krb5kdc[6999](info): commencing operation
krb5kdc: starting...
W20260928 14:51:49.765161 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:49 dist-test-slave-j968 krb5kdc[6999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:49.891032 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:49.973155 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:49.973426 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:49.973465 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:49.974665 7012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:49.974743 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:49.974766 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:49.974786 7012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:49.974817 7012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:49.976372 7012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:49.976667 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:49.976903 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:49.981892 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:49.981892 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:49.982136 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:49.982306 7012 server_base.cc:1034] running on GCE node
I20260928 14:51:49.982555 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:49.982825 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:49.983963 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790607109983937 us; error 45 us; skew 500 ppm
Sep 28 14:51:49 dist-test-slave-j968 krb5kdc[6999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:49.986578 7012 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:49.986972 7012 webserver.cc:460] Webserver started at http://127.0.0.1:34111/ using document root <none> and password file <none>
I20260928 14:51:49.987200 7012 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:49.987246 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:49.987335 7012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:49.988458 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "7d115077fdc14837867549895b85f885"
format_stamp: "Formatted at 2026-09-28 14:51:49 on dist-test-slave-j968"
server_key: "babf542a3bee9abb11d0bb5cdfe141a2"
server_key_iv: "7dddcdfaef6a3abdba34e7e46d423f7a"
server_key_version: "encryptionkey@0"
I20260928 14:51:49.988768 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "7d115077fdc14837867549895b85f885"
format_stamp: "Formatted at 2026-09-28 14:51:49 on dist-test-slave-j968"
server_key: "babf542a3bee9abb11d0bb5cdfe141a2"
server_key_iv: "7dddcdfaef6a3abdba34e7e46d423f7a"
server_key_version: "encryptionkey@0"
I20260928 14:51:49.990046 7012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:51:49.990833 7027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:49.991010 7012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:51:49.991119 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "7d115077fdc14837867549895b85f885"
format_stamp: "Formatted at 2026-09-28 14:51:49 on dist-test-slave-j968"
server_key: "babf542a3bee9abb11d0bb5cdfe141a2"
server_key_iv: "7dddcdfaef6a3abdba34e7e46d423f7a"
server_key_version: "encryptionkey@0"
I20260928 14:51:49.991202 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:50.010121 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:50.012209 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:50.012380 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:50.016501 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37317
I20260928 14:51:50.016541 7079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37317 every 8 connection(s)
I20260928 14:51:50.016917 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:50.017858 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:50.020318 7080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885: Bootstrap starting.
I20260928 14:51:50.020926 7080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:50.021238 7080 log.cc:824] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:50.022006 7080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885: No bootstrap required, opened a new log
I20260928 14:51:50.023700 7080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d115077fdc14837867549895b85f885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } }
I20260928 14:51:50.023844 7080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:50.023922 7080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d115077fdc14837867549895b85f885, State: Initialized, Role: FOLLOWER
I20260928 14:51:50.024120 7080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d115077fdc14837867549895b85f885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } }
I20260928 14:51:50.024192 7080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:50.024236 7080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:50.024290 7080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:50.024833 7080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d115077fdc14837867549895b85f885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } }
I20260928 14:51:50.024948 7080 leader_election.cc:302] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d115077fdc14837867549895b85f885; no voters:
I20260928 14:51:50.025148 7080 leader_election.cc:288] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:50.025209 7083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:50.025370 7083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [term 1 LEADER]: Becoming Leader. State: Replica: 7d115077fdc14837867549895b85f885, State: Running, Role: LEADER
I20260928 14:51:50.025413 7080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:50.025456 7083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d115077fdc14837867549895b85f885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } }
I20260928 14:51:50.025849 7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d115077fdc14837867549895b85f885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d115077fdc14837867549895b85f885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } } }
I20260928 14:51:50.025935 7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:50.026062 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d115077fdc14837867549895b85f885. Latest consensus state: current_term: 1 leader_uuid: "7d115077fdc14837867549895b85f885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d115077fdc14837867549895b85f885" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37317 } } }
I20260928 14:51:50.026165 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:50.026292 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 7012
I20260928 14:51:50.026389 7091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:50.026418 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:50.026677 18818 external_mini_cluster.cc:1428] Setting key 90957e0011c4b0913bfa9176f5cb6b88
I20260928 14:51:50.027175 7091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:50.028898 7091 catalog_manager.cc:1347] Generated new cluster ID: c9b548a487324953b9a40c1d8c4280d2
I20260928 14:51:50.028966 7091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:50.036336 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:50.027326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34903 (local address 127.0.0.1:37317)
0928 14:51:50.027474 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:50.027484 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:50.027670 (+ 186us) server_negotiation.cc:407] Connection header received
0928 14:51:50.027815 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:50.027822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:50.027911 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:50.028073 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:50.028342 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:50.032881 (+ 4539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:50.033988 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:50.034637 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:50.034754 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:50.035480 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:50.035484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:50.035485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:50.035700 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:50.035742 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:50.035757 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:50.035827 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:50.035918 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:51:50.036103 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260928 14:51:50.037227 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:50.043741 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:50.040149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40429 (local address 127.0.0.1:37317)
0928 14:51:50.040192 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:50.040194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:50.040376 (+ 182us) server_negotiation.cc:407] Connection header received
0928 14:51:50.040421 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:50.040422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:50.040433 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:50.040496 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:50.040621 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:50.041392 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:50.042391 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:50.042715 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:50.042787 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:50.043481 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:50.043483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:50.043483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:50.043490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:50.043507 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:50.043511 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:50.043569 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:50.043630 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 14:51:50.043656 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:51:50.047052 7091 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:50.047596 7091 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:50.054510 7091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d115077fdc14837867549895b85f885: Generated new TSK 0
I20260928 14:51:50.054744 7091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:50.063418 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 7012
2026-09-28T14:51:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:50 dist-test-slave-j968 krb5kdc[7111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:50 dist-test-slave-j968 krb5kdc[7111](info): set up 2 sockets
Sep 28 14:51:50 dist-test-slave-j968 krb5kdc[7111](info): commencing operation
krb5kdc: starting...
W20260928 14:51:51.109536 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:51 dist-test-slave-j968 krb5kdc[7111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:51.235209 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:51.316594 7124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:51.316849 7124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:51.316890 7124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:51.318074 7124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:51.318152 7124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:51.318178 7124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:51.318209 7124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:51.318243 7124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:51.319749 7124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:51.320062 7124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:51.320312 7124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:51.325575 7132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:51.325563 7130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:51.325583 7129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:51.325618 7124 server_base.cc:1034] running on GCE node
I20260928 14:51:51.326102 7124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:51.326413 7124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:51.327576 7124 hybrid_clock.cc:648] HybridClock initialized: now 1790607111327547 us; error 48 us; skew 500 ppm
Sep 28 14:51:51 dist-test-slave-j968 krb5kdc[7111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:51.330802 7124 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:51.331400 7124 webserver.cc:460] Webserver started at http://127.0.0.1:35171/ using document root <none> and password file <none>
I20260928 14:51:51.331651 7124 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:51.331720 7124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:51.331928 7124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:51.333101 7124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "eeb0551746a442bf82ef8588a1b7761c"
format_stamp: "Formatted at 2026-09-28 14:51:51 on dist-test-slave-j968"
server_key: "aed05b68bab1c6865de676b553ac3d47"
server_key_iv: "5b47ccc9cda102f02a4d611896f3617c"
server_key_version: "encryptionkey@0"
I20260928 14:51:51.333477 7124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "eeb0551746a442bf82ef8588a1b7761c"
format_stamp: "Formatted at 2026-09-28 14:51:51 on dist-test-slave-j968"
server_key: "aed05b68bab1c6865de676b553ac3d47"
server_key_iv: "5b47ccc9cda102f02a4d611896f3617c"
server_key_version: "encryptionkey@0"
I20260928 14:51:51.334929 7124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:51:51.335819 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:51.336113 7124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:51.336228 7124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "eeb0551746a442bf82ef8588a1b7761c"
format_stamp: "Formatted at 2026-09-28 14:51:51 on dist-test-slave-j968"
server_key: "aed05b68bab1c6865de676b553ac3d47"
server_key_iv: "5b47ccc9cda102f02a4d611896f3617c"
server_key_version: "encryptionkey@0"
I20260928 14:51:51.336313 7124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:51.339833 7124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:51.342182 7124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:51.342342 7124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:51.346616 7124 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42203
I20260928 14:51:51.346661 7191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42203 every 8 connection(s)
I20260928 14:51:51.347019 7124 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:51.347936 7192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:51.349931 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 7124
I20260928 14:51:51.350028 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:51.350231 18818 external_mini_cluster.cc:1428] Setting key 84fa7142909becac77cc5c9f7986176d
I20260928 14:51:51.350337 7192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c: Bootstrap starting.
I20260928 14:51:51.351059 7192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:51.351358 7192 log.cc:824] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:51.352219 7192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c: No bootstrap required, opened a new log
I20260928 14:51:51.352903 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:51.350775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49849 (local address 127.0.0.1:42203)
0928 14:51:51.350968 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:51.350976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:51.351000 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:51:51.351516 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:51.351523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:51.351567 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:51.351746 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:51.352488 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:51.352492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:51.352492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:51.352496 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:51.352545 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:51.352555 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:51.352663 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:51.352695 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 14:51:51.352735 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260928 14:51:51.353935 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:51.354713 7192 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb0551746a442bf82ef8588a1b7761c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42203 } }
I20260928 14:51:51.354880 7192 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:51.354923 7192 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb0551746a442bf82ef8588a1b7761c, State: Initialized, Role: FOLLOWER
I20260928 14:51:51.355049 7192 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb0551746a442bf82ef8588a1b7761c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42203 } }
I20260928 14:51:51.355159 7192 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:51.355204 7192 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:51.355255 7192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:51.356069 7192 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb0551746a442bf82ef8588a1b7761c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42203 } }
I20260928 14:51:51.356205 7192 leader_election.cc:302] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeb0551746a442bf82ef8588a1b7761c; no voters:
I20260928 14:51:51.356377 7192 leader_election.cc:288] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:51.356449 7202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:51.356621 7202 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [term 1 LEADER]: Becoming Leader. State: Replica: eeb0551746a442bf82ef8588a1b7761c, State: Running, Role: LEADER
I20260928 14:51:51.356653 7192 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:51.356740 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:51.355410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53139 (local address 127.0.0.1:42203)
0928 14:51:51.355513 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:51.355515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:51.355642 (+ 127us) server_negotiation.cc:407] Connection header received
0928 14:51:51.355672 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:51.355673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:51.355677 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:51.355734 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:51.356358 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:51.356360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:51.356360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:51.356364 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:51.356378 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:51.356381 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:51.356437 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:51.356633 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 14:51:51.356649 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:51:51.356946 7202 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb0551746a442bf82ef8588a1b7761c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42203 } }
I20260928 14:51:51.357378 7203 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeb0551746a442bf82ef8588a1b7761c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb0551746a442bf82ef8588a1b7761c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42203 } } }
I20260928 14:51:51.357390 7204 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeb0551746a442bf82ef8588a1b7761c. Latest consensus state: current_term: 1 leader_uuid: "eeb0551746a442bf82ef8588a1b7761c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb0551746a442bf82ef8588a1b7761c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42203 } } }
I20260928 14:51:51.357523 7204 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:51.357678 7203 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c [sys.catalog]: This master's current role is: LEADER
W20260928 14:51:51.359054 7218 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:51:51.359117 7218 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:51:51.359192 7211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:51.359390 7211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:51.360847 7211 catalog_manager.cc:1347] Generated new cluster ID: 82fac27c9dfe4af8a5271d4573abb39e
I20260928 14:51:51.360899 7211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:51.376976 7211 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:51.377189 7211 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:51.380717 7211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeb0551746a442bf82ef8588a1b7761c: Generated new TSK 0
I20260928 14:51:51.380887 7211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:51.389276 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 7124
2026-09-28T14:51:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:51 dist-test-slave-j968 krb5kdc[7223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:51 dist-test-slave-j968 krb5kdc[7223](info): set up 2 sockets
Sep 28 14:51:51 dist-test-slave-j968 krb5kdc[7223](info): commencing operation
krb5kdc: starting...
W20260928 14:51:52.437028 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:52 dist-test-slave-j968 krb5kdc[7223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:52.575332 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:52.660835 7236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:52.661052 7236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:52.661075 7236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:52.662256 7236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:52.662299 7236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:52.662314 7236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:52.662326 7236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:52.662338 7236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:52.663781 7236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:52.664005 7236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:52.664188 7236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:52.669188 7242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:52.669214 7241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:52.669219 7236 server_base.cc:1034] running on GCE node
W20260928 14:51:52.669188 7244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:52.669730 7236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:52.670030 7236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:52.671206 7236 hybrid_clock.cc:648] HybridClock initialized: now 1790607112671201 us; error 37 us; skew 500 ppm
Sep 28 14:51:52 dist-test-slave-j968 krb5kdc[7223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:52.673722 7236 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:52.674217 7236 webserver.cc:460] Webserver started at http://127.0.0.1:38925/ using document root <none> and password file <none>
I20260928 14:51:52.674449 7236 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:52.674515 7236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:52.674672 7236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:52.675796 7236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "c4d7cc809dda43b29909a3fa29349ed6"
format_stamp: "Formatted at 2026-09-28 14:51:52 on dist-test-slave-j968"
server_key: "97278068bbf264cff7ff115143aa37b5"
server_key_iv: "dcef89eca6b5b8deef4e31873ca1bf93"
server_key_version: "encryptionkey@0"
I20260928 14:51:52.676242 7236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "c4d7cc809dda43b29909a3fa29349ed6"
format_stamp: "Formatted at 2026-09-28 14:51:52 on dist-test-slave-j968"
server_key: "97278068bbf264cff7ff115143aa37b5"
server_key_iv: "dcef89eca6b5b8deef4e31873ca1bf93"
server_key_version: "encryptionkey@0"
I20260928 14:51:52.677676 7236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:52.678471 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:52.678692 7236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:52.678805 7236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "c4d7cc809dda43b29909a3fa29349ed6"
format_stamp: "Formatted at 2026-09-28 14:51:52 on dist-test-slave-j968"
server_key: "97278068bbf264cff7ff115143aa37b5"
server_key_iv: "dcef89eca6b5b8deef4e31873ca1bf93"
server_key_version: "encryptionkey@0"
I20260928 14:51:52.678879 7236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:52.690169 7236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:52.692284 7236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:52.692451 7236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:52.696746 7236 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44241
I20260928 14:51:52.696786 7303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44241 every 8 connection(s)
I20260928 14:51:52.697199 7236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:52.698092 7304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:52.700603 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6: Bootstrap starting.
I20260928 14:51:52.700907 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 7236
I20260928 14:51:52.701073 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:52.701272 18818 external_mini_cluster.cc:1428] Setting key bd0daa4291d84ee5ddd53b7b69801d9f
I20260928 14:51:52.701364 7304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:52.701695 7304 log.cc:824] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:52.702468 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6: No bootstrap required, opened a new log
I20260928 14:51:52.704979 7304 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7cc809dda43b29909a3fa29349ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } }
I20260928 14:51:52.705166 7304 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:52.705205 7304 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d7cc809dda43b29909a3fa29349ed6, State: Initialized, Role: FOLLOWER
I20260928 14:51:52.705328 7304 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7cc809dda43b29909a3fa29349ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } }
I20260928 14:51:52.705406 7304 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:52.705435 7304 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:52.705471 7304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:52.706128 7304 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7cc809dda43b29909a3fa29349ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } }
I20260928 14:51:52.706221 7304 leader_election.cc:302] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d7cc809dda43b29909a3fa29349ed6; no voters:
I20260928 14:51:52.706439 7304 leader_election.cc:288] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:52.706528 7309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:52.706722 7309 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [term 1 LEADER]: Becoming Leader. State: Replica: c4d7cc809dda43b29909a3fa29349ed6, State: Running, Role: LEADER
I20260928 14:51:52.706766 7304 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:52.706902 7309 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7cc809dda43b29909a3fa29349ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } }
I20260928 14:51:52.707412 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d7cc809dda43b29909a3fa29349ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7cc809dda43b29909a3fa29349ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } } }
I20260928 14:51:52.707525 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:52.707772 7318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:52.707412 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d7cc809dda43b29909a3fa29349ed6. Latest consensus state: current_term: 1 leader_uuid: "c4d7cc809dda43b29909a3fa29349ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d7cc809dda43b29909a3fa29349ed6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44241 } } }
I20260928 14:51:52.708163 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:52.708458 7318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 14:51:52 dist-test-slave-j968 krb5kdc[7223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:51:52.710300 7318 catalog_manager.cc:1347] Generated new cluster ID: 6d86f47c9ca242a68bdc3843e5db4cad
I20260928 14:51:52.710367 7318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:52.713564 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:52.702088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:59189 (local address 127.0.0.1:44241)
0928 14:51:52.702281 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:52.702288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:52.702313 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:51:52.702875 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:52.702883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:52.702933 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:52.703049 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:52.703824 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:52.704754 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:52.705850 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:52.706338 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:52.706452 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:52.709662 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:52.709667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:52.709667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:52.709681 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:52.711745 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:52.712276 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:52.712279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:52.712279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:52.712385 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:52.712639 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:52.712639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:52.712640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:52.712950 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:52.713095 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:52.713259 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 14:51:52.713354 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260928 14:51:52.714862 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:52.718972 7318 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:52.719419 7318 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:52.722364 7318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d7cc809dda43b29909a3fa29349ed6: Generated new TSK 0
I20260928 14:51:52.722506 7318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:52.724434 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:52.716341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:41313 (local address 127.0.0.1:44241)
0928 14:51:52.716375 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:52.716377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:52.716534 (+ 157us) server_negotiation.cc:407] Connection header received
0928 14:51:52.716601 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:52.716602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:52.716611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:52.716665 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:52.717302 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:52.718093 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:52.719323 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:52.719705 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:52.719750 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:52.721329 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:52.721331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:52.721331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:52.721338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:52.723102 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:52.723491 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:52.723492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:52.723493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:52.723554 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:52.723810 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:52.723811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:52.723812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:52.723985 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:52.724104 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:52.724271 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:51:52.724339 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":15}
I20260928 14:51:52.726752 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 7236
W20260928 14:51:52.731441 7221 connection.cc:535] client connection to 127.0.0.1:44241 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44241): Connection reset by peer (error 104)
2026-09-28T14:51:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:52 dist-test-slave-j968 krb5kdc[7335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:52 dist-test-slave-j968 krb5kdc[7335](info): set up 2 sockets
Sep 28 14:51:52 dist-test-slave-j968 krb5kdc[7335](info): commencing operation
krb5kdc: starting...
W20260928 14:51:53.792985 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:53 dist-test-slave-j968 krb5kdc[7335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:53.919276 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:54.002607 7348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:54.002833 7348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:54.002857 7348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:54.004079 7348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:54.004123 7348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:54.004143 7348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:54.004156 7348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:54.004169 7348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:54.005618 7348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:54.005817 7348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:54.005990 7348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:51:54.011034 7348 server_base.cc:1034] running on GCE node
W20260928 14:51:54.010980 7356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:54.011022 7354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:54.011003 7353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:54.011396 7348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:54.011709 7348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:54.012879 7348 hybrid_clock.cc:648] HybridClock initialized: now 1790607114012862 us; error 32 us; skew 500 ppm
Sep 28 14:51:54 dist-test-slave-j968 krb5kdc[7335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:54.019683 7348 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:54.020361 7348 webserver.cc:460] Webserver started at http://127.0.0.1:42983/ using document root <none> and password file <none>
I20260928 14:51:54.020613 7348 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:54.020684 7348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:54.020829 7348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:54.022054 7348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "086a1e7d5fb84288aa1dcee521e81789"
format_stamp: "Formatted at 2026-09-28 14:51:54 on dist-test-slave-j968"
server_key: "848996219f61b9eddafecafce2e9ecd8"
server_key_iv: "2411235a3fe925d7ff282d9e25555b32"
server_key_version: "encryptionkey@0"
I20260928 14:51:54.022511 7348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "086a1e7d5fb84288aa1dcee521e81789"
format_stamp: "Formatted at 2026-09-28 14:51:54 on dist-test-slave-j968"
server_key: "848996219f61b9eddafecafce2e9ecd8"
server_key_iv: "2411235a3fe925d7ff282d9e25555b32"
server_key_version: "encryptionkey@0"
I20260928 14:51:54.023960 7348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:51:54.024799 7363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:54.025004 7348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:54.025107 7348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "086a1e7d5fb84288aa1dcee521e81789"
format_stamp: "Formatted at 2026-09-28 14:51:54 on dist-test-slave-j968"
server_key: "848996219f61b9eddafecafce2e9ecd8"
server_key_iv: "2411235a3fe925d7ff282d9e25555b32"
server_key_version: "encryptionkey@0"
I20260928 14:51:54.025182 7348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:54.035507 7348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:54.038079 7348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:54.038250 7348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:54.042261 7348 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36165
I20260928 14:51:54.042296 7415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36165 every 8 connection(s)
I20260928 14:51:54.042668 7348 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:54.043546 7416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:54.044580 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 7348
I20260928 14:51:54.044723 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:54.044910 18818 external_mini_cluster.cc:1428] Setting key aea3bc0bb54b93c7f0d4e0d6c8c3c6f2
I20260928 14:51:54.046303 7416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789: Bootstrap starting.
I20260928 14:51:54.046948 7416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:54.047262 7416 log.cc:824] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:54.048017 7416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789: No bootstrap required, opened a new log
I20260928 14:51:54.049995 7416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086a1e7d5fb84288aa1dcee521e81789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260928 14:51:54.050154 7416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:54.050203 7416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086a1e7d5fb84288aa1dcee521e81789, State: Initialized, Role: FOLLOWER
I20260928 14:51:54.050381 7416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086a1e7d5fb84288aa1dcee521e81789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260928 14:51:54.050465 7416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:54.050510 7416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:54.050554 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:54.045457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:34041 (local address 127.0.0.1:36165)
0928 14:51:54.045655 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:54.045663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:54.045694 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:51:54.046129 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:54.046136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:54.046185 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:54.046304 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:54.046557 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:54.047594 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:54.048527 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:54.048950 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:54.049068 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:54.049863 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:54.049874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:54.049874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:54.050035 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:54.050065 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:54.050075 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:54.050146 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:54.050303 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:51:54.050336 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260928 14:51:54.050559 7416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:54.051213 7416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086a1e7d5fb84288aa1dcee521e81789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260928 14:51:54.051337 7416 leader_election.cc:302] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 086a1e7d5fb84288aa1dcee521e81789; no voters:
I20260928 14:51:54.051530 7416 leader_election.cc:288] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:54.051523 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:54.051559 7421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:54.051736 7421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [term 1 LEADER]: Becoming Leader. State: Replica: 086a1e7d5fb84288aa1dcee521e81789, State: Running, Role: LEADER
I20260928 14:51:54.051764 7416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:54.051847 7421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086a1e7d5fb84288aa1dcee521e81789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260928 14:51:54.052258 7422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "086a1e7d5fb84288aa1dcee521e81789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086a1e7d5fb84288aa1dcee521e81789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } } }
I20260928 14:51:54.052348 7422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:54.052532 7423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 086a1e7d5fb84288aa1dcee521e81789. Latest consensus state: current_term: 1 leader_uuid: "086a1e7d5fb84288aa1dcee521e81789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086a1e7d5fb84288aa1dcee521e81789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } } }
I20260928 14:51:54.052591 7423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789 [sys.catalog]: This master's current role is: LEADER
W20260928 14:51:54.053781 7440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:51:54.053838 7440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:51:54.053882 7428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:54.054034 7428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:54.055560 7428 catalog_manager.cc:1347] Generated new cluster ID: a051b3fab9f04d97afb44a38d4e7906f
I20260928 14:51:54.055625 7428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:54.056731 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:54.053117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:44189 (local address 127.0.0.1:36165)
0928 14:51:54.053156 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:54.053158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:54.053490 (+ 332us) server_negotiation.cc:407] Connection header received
0928 14:51:54.053561 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:54.053561 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:54.053571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:54.053649 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:54.053811 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:54.054408 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:54.055334 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:54.055652 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:54.055716 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:54.056491 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:54.056493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:54.056493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:54.056498 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:54.056513 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:54.056517 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:54.056573 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:54.056647 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 14:51:54.056657 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:51:54.063578 7428 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:54.064357 7428 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:54.070636 7428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 086a1e7d5fb84288aa1dcee521e81789: Generated new TSK 0
I20260928 14:51:54.070835 7428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:54.075659 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 7348
W20260928 14:51:54.079860 7333 connection.cc:535] client connection to 127.0.0.1:36165 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36165): Connection reset by peer (error 104)
2026-09-28T14:51:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:54 dist-test-slave-j968 krb5kdc[7447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:54 dist-test-slave-j968 krb5kdc[7447](info): set up 2 sockets
Sep 28 14:51:54 dist-test-slave-j968 krb5kdc[7447](info): commencing operation
krb5kdc: starting...
W20260928 14:51:55.172946 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:55 dist-test-slave-j968 krb5kdc[7447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:55.300376 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:55.387418 7460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:55.387698 7460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:55.387741 7460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:55.388989 7460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:55.389071 7460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:55.389096 7460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:55.389119 7460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:55.389223 7460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:55.390794 7460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:55.391093 7460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:55.391343 7460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:55.396389 7465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:55.396456 7468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:55.396373 7466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:55.396674 7460 server_base.cc:1034] running on GCE node
I20260928 14:51:55.396908 7460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:55.397256 7460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:55.398430 7460 hybrid_clock.cc:648] HybridClock initialized: now 1790607115398410 us; error 39 us; skew 500 ppm
Sep 28 14:51:55 dist-test-slave-j968 krb5kdc[7447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:55.401239 7460 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:55.401834 7460 webserver.cc:460] Webserver started at http://127.0.0.1:45421/ using document root <none> and password file <none>
I20260928 14:51:55.402186 7460 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:55.402266 7460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:55.402415 7460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:55.403501 7460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "a195fc1039be432698f02bf08aafa5fb"
format_stamp: "Formatted at 2026-09-28 14:51:55 on dist-test-slave-j968"
server_key: "f7872a0c244b3191523f207743afacda"
server_key_iv: "fb33f47de0f621dae537e5e55a103cc5"
server_key_version: "encryptionkey@0"
I20260928 14:51:55.403807 7460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "a195fc1039be432698f02bf08aafa5fb"
format_stamp: "Formatted at 2026-09-28 14:51:55 on dist-test-slave-j968"
server_key: "f7872a0c244b3191523f207743afacda"
server_key_iv: "fb33f47de0f621dae537e5e55a103cc5"
server_key_version: "encryptionkey@0"
I20260928 14:51:55.405265 7460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:55.406105 7475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:55.406561 7460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:51:55.406669 7460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "a195fc1039be432698f02bf08aafa5fb"
format_stamp: "Formatted at 2026-09-28 14:51:55 on dist-test-slave-j968"
server_key: "f7872a0c244b3191523f207743afacda"
server_key_iv: "fb33f47de0f621dae537e5e55a103cc5"
server_key_version: "encryptionkey@0"
I20260928 14:51:55.406751 7460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:55.410454 7460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:55.412648 7460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:55.412761 7460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:55.416708 7460 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46169
I20260928 14:51:55.416739 7527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46169 every 8 connection(s)
I20260928 14:51:55.417098 7460 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:55.417999 7528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:55.420428 7528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb: Bootstrap starting.
I20260928 14:51:55.421023 7528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:55.421329 7528 log.cc:824] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:55.422091 7528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb: No bootstrap required, opened a new log
I20260928 14:51:55.423765 7528 raft_consensus.cc:348] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195fc1039be432698f02bf08aafa5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260928 14:51:55.423983 7528 raft_consensus.cc:374] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:55.424013 7528 raft_consensus.cc:729] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a195fc1039be432698f02bf08aafa5fb, State: Initialized, Role: FOLLOWER
I20260928 14:51:55.424186 7528 consensus_queue.cc:260] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195fc1039be432698f02bf08aafa5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260928 14:51:55.424250 7528 raft_consensus.cc:388] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:55.424301 7528 raft_consensus.cc:482] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:55.424355 7528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:55.424932 7528 raft_consensus.cc:504] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195fc1039be432698f02bf08aafa5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260928 14:51:55.425019 7528 leader_election.cc:302] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a195fc1039be432698f02bf08aafa5fb; no voters:
I20260928 14:51:55.425206 7528 leader_election.cc:288] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:55.425276 7531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:55.425478 7528 sys_catalog.cc:564] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:55.425508 7531 raft_consensus.cc:686] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [term 1 LEADER]: Becoming Leader. State: Replica: a195fc1039be432698f02bf08aafa5fb, State: Running, Role: LEADER
I20260928 14:51:55.425688 7531 consensus_queue.cc:237] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195fc1039be432698f02bf08aafa5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260928 14:51:55.425879 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 7460
I20260928 14:51:55.426002 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:55.426213 18818 external_mini_cluster.cc:1428] Setting key ddad00260e611bbb78150a5d698586f0
I20260928 14:51:55.427059 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a195fc1039be432698f02bf08aafa5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195fc1039be432698f02bf08aafa5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } } }
I20260928 14:51:55.427206 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:55.427428 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a195fc1039be432698f02bf08aafa5fb. Latest consensus state: current_term: 1 leader_uuid: "a195fc1039be432698f02bf08aafa5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a195fc1039be432698f02bf08aafa5fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } } }
I20260928 14:51:55.427500 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb [sys.catalog]: This master's current role is: LEADER
W20260928 14:51:55.427588 7547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:51:55.427634 7547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:51:55.427718 7549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:55.427991 7549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:55.429095 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:55.426729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:34477 (local address 127.0.0.1:46169)
0928 14:51:55.426912 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:55.426923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:55.427125 (+ 202us) server_negotiation.cc:407] Connection header received
0928 14:51:55.427357 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:55.427365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:55.427396 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:55.427531 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:55.428632 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:55.428636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:55.428637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:55.428641 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:55.428689 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:55.428705 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:55.428780 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:55.428862 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 14:51:55.428906 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260928 14:51:55.429672 7549 catalog_manager.cc:1347] Generated new cluster ID: 105715c4f6174866a339ab7091e31711
I20260928 14:51:55.429719 7549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:55.430202 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:55.432818 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:55.431784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:41257 (local address 127.0.0.1:46169)
0928 14:51:55.431818 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:55.431820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:55.431828 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:51:55.431906 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:55.431907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:55.431913 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:55.431974 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:55.432589 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:55.432590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:55.432590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:55.432594 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:55.432611 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:55.432615 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:55.432678 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:55.432729 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:51:55.432743 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 14:51:55.438251 7549 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:55.438445 7549 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:55.447240 7549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a195fc1039be432698f02bf08aafa5fb: Generated new TSK 0
I20260928 14:51:55.447463 7549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:55.451666 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 7460
2026-09-28T14:51:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:55 dist-test-slave-j968 krb5kdc[7559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:55 dist-test-slave-j968 krb5kdc[7559](info): set up 2 sockets
Sep 28 14:51:55 dist-test-slave-j968 krb5kdc[7559](info): commencing operation
krb5kdc: starting...
W20260928 14:51:56.529014 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:56 dist-test-slave-j968 krb5kdc[7559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:56.651947 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36541
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:56.733696 7572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:56.733911 7572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:56.733935 7572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:56.735040 7572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:56.735078 7572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:56.735092 7572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:56.735105 7572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:56.735116 7572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:56.736644 7572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36541
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:56.736857 7572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:56.737039 7572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:56.742188 7578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:56.742197 7577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:56.742178 7580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:56.742419 7572 server_base.cc:1034] running on GCE node
I20260928 14:51:56.742671 7572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:56.743011 7572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:56.744181 7572 hybrid_clock.cc:648] HybridClock initialized: now 1790607116744147 us; error 53 us; skew 500 ppm
Sep 28 14:51:56 dist-test-slave-j968 krb5kdc[7559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:56.747182 7572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:56.747666 7572 webserver.cc:460] Webserver started at http://127.0.0.1:42071/ using document root <none> and password file <none>
I20260928 14:51:56.747939 7572 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:56.747988 7572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:56.748082 7572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:56.749116 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19"
format_stamp: "Formatted at 2026-09-28 14:51:56 on dist-test-slave-j968"
server_key: "f6e050976e7fb430f5f27c23e846f274"
server_key_iv: "eb464e8b9ca1931b91a3f4554a483001"
server_key_version: "encryptionkey@0"
I20260928 14:51:56.749471 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19"
format_stamp: "Formatted at 2026-09-28 14:51:56 on dist-test-slave-j968"
server_key: "f6e050976e7fb430f5f27c23e846f274"
server_key_iv: "eb464e8b9ca1931b91a3f4554a483001"
server_key_version: "encryptionkey@0"
I20260928 14:51:56.750788 7572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:51:56.751632 7587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:56.751812 7572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:56.751928 7572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19"
format_stamp: "Formatted at 2026-09-28 14:51:56 on dist-test-slave-j968"
server_key: "f6e050976e7fb430f5f27c23e846f274"
server_key_iv: "eb464e8b9ca1931b91a3f4554a483001"
server_key_version: "encryptionkey@0"
I20260928 14:51:56.752005 7572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:56.766345 7572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:56.768715 7572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:56.768838 7572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:56.773042 7572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36541
I20260928 14:51:56.773109 7639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36541 every 8 connection(s)
I20260928 14:51:56.773478 7572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:56.774510 7640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:56.777040 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 7572
I20260928 14:51:56.777078 7640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19: Bootstrap starting.
I20260928 14:51:56.777154 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:56.777329 18818 external_mini_cluster.cc:1428] Setting key dcca7abd44559e1adfd85609c26cd85e
I20260928 14:51:56.777812 7640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:56.778196 7640 log.cc:824] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:56.779007 7640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19: No bootstrap required, opened a new log
I20260928 14:51:56.781378 7640 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36541 } }
I20260928 14:51:56.781580 7640 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:56.781632 7640 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6ddf1d602c44ac9d3c2b5f6c281a19, State: Initialized, Role: FOLLOWER
I20260928 14:51:56.781810 7640 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36541 } }
I20260928 14:51:56.781906 7640 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:56.781952 7640 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:56.782015 7640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:56.782739 7640 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36541 } }
I20260928 14:51:56.782850 7640 leader_election.cc:302] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf6ddf1d602c44ac9d3c2b5f6c281a19; no voters:
I20260928 14:51:56.783017 7640 leader_election.cc:288] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:56.783159 7645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:56.783301 7640 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:56.783365 7645 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [term 1 LEADER]: Becoming Leader. State: Replica: cf6ddf1d602c44ac9d3c2b5f6c281a19, State: Running, Role: LEADER
I20260928 14:51:56.783581 7645 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36541 } }
I20260928 14:51:56.784065 7646 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36541 } } }
I20260928 14:51:56.784089 7647 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf6ddf1d602c44ac9d3c2b5f6c281a19. Latest consensus state: current_term: 1 leader_uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6ddf1d602c44ac9d3c2b5f6c281a19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36541 } } }
I20260928 14:51:56.784205 7646 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:56.784219 7647 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19 [sys.catalog]: This master's current role is: LEADER
Sep 28 14:51:56 dist-test-slave-j968 krb5kdc[7559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260928 14:51:56.785720 7661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:51:56.785811 7661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:51:56.785872 7651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:56.786088 7651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:56.787719 7651 catalog_manager.cc:1347] Generated new cluster ID: 3c5e7645177149e9be0a49b72bd72aa9
I20260928 14:51:56.787765 7651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:56.789870 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:56.777957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:43815 (local address 127.0.0.1:36541)
0928 14:51:56.778232 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:56.778242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:56.778274 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:51:56.778881 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:56.778890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:56.778958 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:56.779116 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:56.779834 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:56.781334 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:56.782425 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:56.783090 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:56.783247 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:56.786270 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:56.786274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:56.786275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:56.786287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:56.788258 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:56.788672 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:56.788676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:56.788676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:56.788768 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:56.789003 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:56.789004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:56.789005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:56.789256 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:56.789410 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:56.789540 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 14:51:56.789640 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":140,"threads_started":1}
I20260928 14:51:56.790899 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:56.798336 7651 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:56.799046 7651 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:56.802971 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:56.796350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:51931 (local address 127.0.0.1:36541)
0928 14:51:56.796394 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:56.796396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:56.796586 (+ 190us) server_negotiation.cc:407] Connection header received
0928 14:51:56.796673 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:56.796674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:56.796684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:56.796733 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:56.797325 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:56.797903 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:56.798772 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:56.798990 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:56.799076 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:56.800378 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:56.800379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:56.800380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:51:56.800404 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:56.801805 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:56.802194 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:56.802195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:56.802196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:56.802251 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:51:56.802469 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:51:56.802470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:51:56.802470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:51:56.802574 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:56.802685 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:56.802806 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 14:51:56.802869 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:51:56.805909 7651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf6ddf1d602c44ac9d3c2b5f6c281a19: Generated new TSK 0
I20260928 14:51:56.806087 7651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:56.809193 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 7572
W20260928 14:51:56.813756 7557 connection.cc:535] client connection to 127.0.0.1:36541 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36541): Connection reset by peer (error 104)
2026-09-28T14:51:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:56 dist-test-slave-j968 krb5kdc[7671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:56 dist-test-slave-j968 krb5kdc[7671](info): set up 2 sockets
Sep 28 14:51:56 dist-test-slave-j968 krb5kdc[7671](info): commencing operation
krb5kdc: starting...
W20260928 14:51:57.853371 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:57 dist-test-slave-j968 krb5kdc[7671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:57.979470 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:58.066175 7684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:58.066442 7684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:58.066481 7684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:58.067670 7684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:58.067751 7684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:58.067775 7684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:58.067797 7684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:58.067826 7684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:58.069396 7684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:58.069693 7684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:58.069933 7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:58.075529 7690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:58.075616 7684 server_base.cc:1034] running on GCE node
W20260928 14:51:58.075531 7692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:58.075573 7689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:58.076030 7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:58.076292 7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:58.077450 7684 hybrid_clock.cc:648] HybridClock initialized: now 1790607118077408 us; error 56 us; skew 500 ppm
Sep 28 14:51:58 dist-test-slave-j968 krb5kdc[7671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:58.080139 7684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:58.080639 7684 webserver.cc:460] Webserver started at http://127.0.0.1:37557/ using document root <none> and password file <none>
I20260928 14:51:58.080875 7684 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:58.080945 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:58.081084 7684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:58.082240 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f"
format_stamp: "Formatted at 2026-09-28 14:51:58 on dist-test-slave-j968"
server_key: "23631119c280be4375f4b848c475b7c0"
server_key_iv: "5beb34dfb947d19866df926dba4eb622"
server_key_version: "encryptionkey@0"
I20260928 14:51:58.082630 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f"
format_stamp: "Formatted at 2026-09-28 14:51:58 on dist-test-slave-j968"
server_key: "23631119c280be4375f4b848c475b7c0"
server_key_iv: "5beb34dfb947d19866df926dba4eb622"
server_key_version: "encryptionkey@0"
I20260928 14:51:58.084115 7684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:51:58.084910 7699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:58.085239 7684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:58.085345 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f"
format_stamp: "Formatted at 2026-09-28 14:51:58 on dist-test-slave-j968"
server_key: "23631119c280be4375f4b848c475b7c0"
server_key_iv: "5beb34dfb947d19866df926dba4eb622"
server_key_version: "encryptionkey@0"
I20260928 14:51:58.085409 7684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:58.112502 7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:58.115231 7684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:58.115355 7684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:58.119699 7684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35945
I20260928 14:51:58.119779 7751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35945 every 8 connection(s)
I20260928 14:51:58.120162 7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:58.121116 7752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:58.123584 7752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f: Bootstrap starting.
I20260928 14:51:58.124266 7752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:58.124572 7752 log.cc:824] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:58.125309 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 7684
I20260928 14:51:58.125434 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:58.125619 7752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f: No bootstrap required, opened a new log
I20260928 14:51:58.125653 18818 external_mini_cluster.cc:1428] Setting key 09493b33e8aa94695fde9262ee5f9dea
I20260928 14:51:58.128705 7752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35945 } }
I20260928 14:51:58.128859 7752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:58.128887 7752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd9b1c1e2724c628cdd035a0a3ed23f, State: Initialized, Role: FOLLOWER
I20260928 14:51:58.129038 7752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35945 } }
I20260928 14:51:58.129103 7752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:58.129125 7752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:58.129194 7752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:58.129838 7752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35945 } }
I20260928 14:51:58.129966 7752 leader_election.cc:302] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cd9b1c1e2724c628cdd035a0a3ed23f; no voters:
I20260928 14:51:58.130193 7752 leader_election.cc:288] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:58.130215 7757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:58.130417 7757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [term 1 LEADER]: Becoming Leader. State: Replica: 7cd9b1c1e2724c628cdd035a0a3ed23f, State: Running, Role: LEADER
I20260928 14:51:58.130548 7757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35945 } }
I20260928 14:51:58.130877 7757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cd9b1c1e2724c628cdd035a0a3ed23f. Latest consensus state: current_term: 1 leader_uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35945 } } }
I20260928 14:51:58.130891 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cd9b1c1e2724c628cdd035a0a3ed23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35945 } } }
I20260928 14:51:58.130970 7757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:58.131188 7761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:58.130545 7752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:58.130975 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:58.132005 7761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:58.134805 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:58.126401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:50613 (local address 127.0.0.1:35945)
0928 14:51:58.126611 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:58.126620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:58.126647 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:51:58.127209 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:58.127217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:58.127273 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:58.127461 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:58.127644 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:58.128608 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:58.129618 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:58.130153 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:58.130299 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:58.131072 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:58.131077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:58.131077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:58.131302 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:58.131344 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:58.131370 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:58.131469 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:58.131611 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 14:51:58.134453 (+ 2842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260928 14:51:58.136030 7761 catalog_manager.cc:1347] Generated new cluster ID: 0b774a1f516d4cafb50428ec89bce5ae
I20260928 14:51:58.136113 7761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:58.138645 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:58.146102 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:58.140499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:39587 (local address 127.0.0.1:35945)
0928 14:51:58.140541 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:58.140543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:58.140639 (+ 96us) server_negotiation.cc:407] Connection header received
0928 14:51:58.140714 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:58.140715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:58.140726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:58.140792 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:58.142564 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:58.143293 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:58.144498 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:51:58.144840 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:51:58.144917 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:51:58.145823 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:58.145825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:58.145825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:58.145831 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:58.145849 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:58.145853 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:58.145919 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:58.146003 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 14:51:58.146016 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:51:58.164135 7761 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:58.164921 7761 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:58.176631 7761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cd9b1c1e2724c628cdd035a0a3ed23f: Generated new TSK 0
I20260928 14:51:58.176865 7761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:58.182931 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 7684
W20260928 14:51:58.187530 7669 connection.cc:535] client connection to 127.0.0.1:35945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35945): Connection reset by peer (error 104)
2026-09-28T14:51:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:58 dist-test-slave-j968 krb5kdc[7783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:58 dist-test-slave-j968 krb5kdc[7783](info): set up 2 sockets
Sep 28 14:51:58 dist-test-slave-j968 krb5kdc[7783](info): commencing operation
krb5kdc: starting...
W20260928 14:51:59.241513 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:51:59 dist-test-slave-j968 krb5kdc[7783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:51:59.371146 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:51:59.456336 7796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:51:59.456589 7796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:51:59.456629 7796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:51:59.457824 7796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:51:59.457896 7796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:51:59.457921 7796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:51:59.457952 7796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:51:59.458001 7796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:51:59.459537 7796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:51:59.459828 7796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:51:59.460093 7796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:51:59.465157 7802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:59.465206 7801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:51:59.465163 7804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:51:59.465210 7796 server_base.cc:1034] running on GCE node
I20260928 14:51:59.465739 7796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:51:59.466101 7796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:51:59.467278 7796 hybrid_clock.cc:648] HybridClock initialized: now 1790607119467266 us; error 48 us; skew 500 ppm
Sep 28 14:51:59 dist-test-slave-j968 krb5kdc[7783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:51:59.474406 7796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:51:59.474983 7796 webserver.cc:460] Webserver started at http://127.0.0.1:42233/ using document root <none> and password file <none>
I20260928 14:51:59.475286 7796 fs_manager.cc:359] Metadata directory not provided
I20260928 14:51:59.475374 7796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:51:59.475529 7796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:51:59.476805 7796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "a1621d0446ff4bf5af0f100506806576"
format_stamp: "Formatted at 2026-09-28 14:51:59 on dist-test-slave-j968"
server_key: "72b014021a2d6ebadabecfe65b9077ae"
server_key_iv: "cd3483bb2c29d24b25a85d4c4de49585"
server_key_version: "encryptionkey@0"
I20260928 14:51:59.477186 7796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "a1621d0446ff4bf5af0f100506806576"
format_stamp: "Formatted at 2026-09-28 14:51:59 on dist-test-slave-j968"
server_key: "72b014021a2d6ebadabecfe65b9077ae"
server_key_iv: "cd3483bb2c29d24b25a85d4c4de49585"
server_key_version: "encryptionkey@0"
I20260928 14:51:59.478608 7796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 14:51:59.479521 7811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:51:59.479766 7796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:51:59.479938 7796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "a1621d0446ff4bf5af0f100506806576"
format_stamp: "Formatted at 2026-09-28 14:51:59 on dist-test-slave-j968"
server_key: "72b014021a2d6ebadabecfe65b9077ae"
server_key_iv: "cd3483bb2c29d24b25a85d4c4de49585"
server_key_version: "encryptionkey@0"
I20260928 14:51:59.480041 7796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:51:59.483584 7796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:51:59.485730 7796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:51:59.485893 7796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:51:59.490198 7796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42879
I20260928 14:51:59.490231 7863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42879 every 8 connection(s)
I20260928 14:51:59.490579 7796 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:51:59.491521 7864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:51:59.494031 7864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576: Bootstrap starting.
I20260928 14:51:59.494658 7864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576: Neither blocks nor log segments found. Creating new log.
I20260928 14:51:59.494963 7864 log.cc:824] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576: Log is configured to *not* fsync() on all Append() calls
I20260928 14:51:59.495764 7864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576: No bootstrap required, opened a new log
I20260928 14:51:59.496587 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 7796
I20260928 14:51:59.496750 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:51:59.496960 18818 external_mini_cluster.cc:1428] Setting key 589a3e2830074490f094e5cc71ba5d84
I20260928 14:51:59.498325 7864 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1621d0446ff4bf5af0f100506806576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260928 14:51:59.498533 7864 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:51:59.498579 7864 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1621d0446ff4bf5af0f100506806576, State: Initialized, Role: FOLLOWER
I20260928 14:51:59.498791 7864 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1621d0446ff4bf5af0f100506806576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260928 14:51:59.498896 7864 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:51:59.498948 7864 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:51:59.499039 7864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:51:59.499993 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:59.497666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:56353 (local address 127.0.0.1:42879)
0928 14:51:59.497867 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:59.497876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:59.497904 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:51:59.498298 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:59.498307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:59.498347 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:59.498500 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:59.499375 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:59.499379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:59.499379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:59.499384 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:59.499438 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:59.499481 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:59.499565 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:59.499637 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 14:51:59.499704 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260928 14:51:59.500460 7864 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1621d0446ff4bf5af0f100506806576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260928 14:51:59.500572 7864 leader_election.cc:302] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1621d0446ff4bf5af0f100506806576; no voters:
I20260928 14:51:59.500747 7864 leader_election.cc:288] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:51:59.500779 7869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:51:59.500945 7869 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [term 1 LEADER]: Becoming Leader. State: Replica: a1621d0446ff4bf5af0f100506806576, State: Running, Role: LEADER
I20260928 14:51:59.501008 7864 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:51:59.501044 7869 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1621d0446ff4bf5af0f100506806576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260928 14:51:59.501116 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:51:59.501371 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1621d0446ff4bf5af0f100506806576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1621d0446ff4bf5af0f100506806576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } } }
I20260928 14:51:59.501418 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:59.501557 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1621d0446ff4bf5af0f100506806576. Latest consensus state: current_term: 1 leader_uuid: "a1621d0446ff4bf5af0f100506806576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1621d0446ff4bf5af0f100506806576" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } } }
I20260928 14:51:59.501605 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576 [sys.catalog]: This master's current role is: LEADER
I20260928 14:51:59.501608 7875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:51:59.502171 7875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:51:59.504328 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:51:59.502991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:41125 (local address 127.0.0.1:42879)
0928 14:51:59.503032 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:51:59.503034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:51:59.503243 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:51:59.503295 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:51:59.503296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:51:59.503302 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:51:59.503362 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:51:59.504047 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:51:59.504049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:51:59.504049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:51:59.504053 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:51:59.504070 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:51:59.504075 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:51:59.504135 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:51:59.504235 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:51:59.504250 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:51:59.505152 7875 catalog_manager.cc:1347] Generated new cluster ID: d710d33e73f74d50977f0ca559723daa
I20260928 14:51:59.505224 7875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:51:59.511799 7875 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:51:59.512044 7875 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:51:59.515529 7875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1621d0446ff4bf5af0f100506806576: Generated new TSK 0
I20260928 14:51:59.515754 7875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:51:59.518695 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 7796
2026-09-28T14:51:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:51:59 dist-test-slave-j968 krb5kdc[7895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:51:59 dist-test-slave-j968 krb5kdc[7895](info): set up 2 sockets
Sep 28 14:51:59 dist-test-slave-j968 krb5kdc[7895](info): commencing operation
krb5kdc: starting...
W20260928 14:52:00.573122 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:52:00 dist-test-slave-j968 krb5kdc[7895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:52:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:52:00.701073 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34523
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:34523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:52:00.784906 7908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:00.785140 7908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:00.785166 7908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:00.786571 7908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:00.786635 7908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:00.786649 7908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:00.786664 7908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:00.786675 7908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:00.788241 7908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:34523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:00.788452 7908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:00.788652 7908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:00.793799 7913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:00.793812 7916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:00.793831 7914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:00.793843 7908 server_base.cc:1034] running on GCE node
I20260928 14:52:00.794351 7908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:00.794720 7908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:00.795945 7908 hybrid_clock.cc:648] HybridClock initialized: now 1790607120795927 us; error 50 us; skew 500 ppm
Sep 28 14:52:00 dist-test-slave-j968 krb5kdc[7895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607120, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:00.799018 7908 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:52:00.799482 7908 webserver.cc:460] Webserver started at http://127.18.96.190:34117/ using document root <none> and password file <none>
I20260928 14:52:00.799726 7908 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:00.799775 7908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:00.799944 7908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:00.801115 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "ed3ded9f55db4f0f9348fa11efc67f49"
format_stamp: "Formatted at 2026-09-28 14:52:00 on dist-test-slave-j968"
server_key: "f61e61da39e9e875bd9cd569182d559e"
server_key_iv: "4a836893089dd981d2fe77af6b3aa3db"
server_key_version: "encryptionkey@0"
I20260928 14:52:00.801448 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "ed3ded9f55db4f0f9348fa11efc67f49"
format_stamp: "Formatted at 2026-09-28 14:52:00 on dist-test-slave-j968"
server_key: "f61e61da39e9e875bd9cd569182d559e"
server_key_iv: "4a836893089dd981d2fe77af6b3aa3db"
server_key_version: "encryptionkey@0"
I20260928 14:52:00.802884 7908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:52:00.803656 7923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:00.803858 7908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:52:00.804003 7908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "ed3ded9f55db4f0f9348fa11efc67f49"
format_stamp: "Formatted at 2026-09-28 14:52:00 on dist-test-slave-j968"
server_key: "f61e61da39e9e875bd9cd569182d559e"
server_key_iv: "4a836893089dd981d2fe77af6b3aa3db"
server_key_version: "encryptionkey@0"
I20260928 14:52:00.804064 7908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:00.811054 7908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:00.813252 7908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:00.813366 7908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:00.817610 7908 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:34523
I20260928 14:52:00.817682 7975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:34523 every 8 connection(s)
I20260928 14:52:00.818014 7908 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:52:00.818961 7976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:00.821815 7976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49: Bootstrap starting.
I20260928 14:52:00.822497 7976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:00.822835 7976 log.cc:824] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:00.823637 7976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49: No bootstrap required, opened a new log
I20260928 14:52:00.825511 7976 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3ded9f55db4f0f9348fa11efc67f49" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34523 } }
I20260928 14:52:00.825708 7976 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:52:00.825757 7976 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ded9f55db4f0f9348fa11efc67f49, State: Initialized, Role: FOLLOWER
I20260928 14:52:00.825941 7976 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3ded9f55db4f0f9348fa11efc67f49" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34523 } }
I20260928 14:52:00.826047 7976 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:52:00.826095 7976 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:52:00.826148 7976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:00.826472 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 7908
I20260928 14:52:00.826565 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:52:00.826800 18818 external_mini_cluster.cc:1428] Setting key dc344bf013c3c25f97b6ff4332077fb4
I20260928 14:52:00.827309 7976 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3ded9f55db4f0f9348fa11efc67f49" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34523 } }
I20260928 14:52:00.827461 7976 leader_election.cc:302] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed3ded9f55db4f0f9348fa11efc67f49; no voters:
I20260928 14:52:00.827621 7976 leader_election.cc:288] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:52:00.827649 7981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:52:00.827777 7981 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ded9f55db4f0f9348fa11efc67f49, State: Running, Role: LEADER
I20260928 14:52:00.827934 7976 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:00.827971 7981 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3ded9f55db4f0f9348fa11efc67f49" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34523 } }
I20260928 14:52:00.828317 7983 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed3ded9f55db4f0f9348fa11efc67f49. Latest consensus state: current_term: 1 leader_uuid: "ed3ded9f55db4f0f9348fa11efc67f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3ded9f55db4f0f9348fa11efc67f49" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34523 } } }
I20260928 14:52:00.828418 7983 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:00.828673 7985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:52:00.829257 7985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:52:00.829737 7982 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed3ded9f55db4f0f9348fa11efc67f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3ded9f55db4f0f9348fa11efc67f49" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34523 } } }
I20260928 14:52:00.830330 7982 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:00.830960 7985 catalog_manager.cc:1347] Generated new cluster ID: 21351b41d02d4a62a8acdd73cac80669
I20260928 14:52:00.831032 7985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 14:52:00 dist-test-slave-j968 krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:52:00.840118 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:00.827333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45153 (local address 127.18.96.190:34523)
0928 14:52:00.827550 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:00.827562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:00.827597 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:52:00.827727 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:00.827737 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:00.827803 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:00.828009 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:00.828920 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:00.830319 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:00.831379 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:00.832012 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:00.832120 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:00.835944 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:00.835950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:00.835950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:00.835964 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:00.838139 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:00.838645 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:00.838650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:00.838650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:00.838819 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:00.839067 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:00.839068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:00.839068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:00.839431 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:00.839632 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:00.839736 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:52:00.839848 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260928 14:52:00.841576 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:52:00.850618 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:00.843067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44967 (local address 127.18.96.190:34523)
0928 14:52:00.843108 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:00.843110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:00.843288 (+ 178us) server_negotiation.cc:407] Connection header received
0928 14:52:00.843322 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:00.843324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:00.843334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:00.843403 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:00.844045 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:00.844920 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:00.845896 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:00.846249 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:00.846322 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:00.847735 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:00.847737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:00.847738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:00.847745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:00.849383 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:00.849800 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:00.849801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:00.849802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:00.849864 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:00.850092 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:00.850093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:00.850094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:00.850231 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:00.850338 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:00.850454 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:52:00.850520 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:52:00.859815 7985 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:52:00.860401 7985 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:52:00.865762 7985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed3ded9f55db4f0f9348fa11efc67f49: Generated new TSK 0
I20260928 14:52:00.865983 7985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:52:00.873440 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 7908
W20260928 14:52:00.878351 7893 connection.cc:535] client connection to 127.18.96.190:34523 recv error: Network error: failed to read from TLS socket (remote: 127.18.96.190:34523): Connection reset by peer (error 104)
2026-09-28T14:52:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:52:00 dist-test-slave-j968 krb5kdc[8007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:52:00 dist-test-slave-j968 krb5kdc[8007](info): set up 2 sockets
Sep 28 14:52:00 dist-test-slave-j968 krb5kdc[8007](info): commencing operation
krb5kdc: starting...
W20260928 14:52:01.929139 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:52:02 dist-test-slave-j968 krb5kdc[8007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:52:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:52:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:52:02.055822 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:36659
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:42255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:36659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:52:02.147006 8020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:02.147274 8020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:02.147322 8020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:02.148743 8020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:02.148819 8020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:02.148849 8020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:02.148885 8020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:02.148914 8020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:02.150621 8020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:42255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:36659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:36659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:02.150936 8020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:02.151196 8020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:02.156481 8025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:02.156457 8026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:02.156445 8028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:02.156759 8020 server_base.cc:1034] running on GCE node
I20260928 14:52:02.157034 8020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:02.157414 8020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:02.158633 8020 hybrid_clock.cc:648] HybridClock initialized: now 1790607122158601 us; error 57 us; skew 500 ppm
Sep 28 14:52:02 dist-test-slave-j968 krb5kdc[8007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607122, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:02.161828 8020 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:52:02.162420 8020 webserver.cc:460] Webserver started at http://127.18.96.190:41903/ using document root <none> and password file <none>
I20260928 14:52:02.162693 8020 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:02.162774 8020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:02.162943 8020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:02.164285 8020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "80c3019f975c4f1982dd7a4256d275b4"
format_stamp: "Formatted at 2026-09-28 14:52:02 on dist-test-slave-j968"
server_key: "0d1cc7446fe3f1393a681d96a391f4e5"
server_key_iv: "79dc4f33a18fd9368cf1f091de0605b9"
server_key_version: "encryptionkey@0"
I20260928 14:52:02.164745 8020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "80c3019f975c4f1982dd7a4256d275b4"
format_stamp: "Formatted at 2026-09-28 14:52:02 on dist-test-slave-j968"
server_key: "0d1cc7446fe3f1393a681d96a391f4e5"
server_key_iv: "79dc4f33a18fd9368cf1f091de0605b9"
server_key_version: "encryptionkey@0"
I20260928 14:52:02.166365 8020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:52:02.167399 8035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:02.167734 8020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:52:02.167905 8020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "80c3019f975c4f1982dd7a4256d275b4"
format_stamp: "Formatted at 2026-09-28 14:52:02 on dist-test-slave-j968"
server_key: "0d1cc7446fe3f1393a681d96a391f4e5"
server_key_iv: "79dc4f33a18fd9368cf1f091de0605b9"
server_key_version: "encryptionkey@0"
I20260928 14:52:02.168015 8020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:02.185346 8020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:02.188159 8020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:02.188356 8020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:02.193161 8020 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:36659
I20260928 14:52:02.193248 8087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:36659 every 8 connection(s)
I20260928 14:52:02.193665 8020 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:52:02.194712 8088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:02.197674 8088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4: Bootstrap starting.
I20260928 14:52:02.198385 8088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:02.198740 8088 log.cc:824] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:02.199676 8088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4: No bootstrap required, opened a new log
I20260928 14:52:02.201546 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 8020
I20260928 14:52:02.201694 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:52:02.201949 18818 external_mini_cluster.cc:1428] Setting key 2736ed6e45c9db13104237bc89bbdecf
I20260928 14:52:02.202517 8088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c3019f975c4f1982dd7a4256d275b4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36659 } }
I20260928 14:52:02.202754 8088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:52:02.202845 8088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c3019f975c4f1982dd7a4256d275b4, State: Initialized, Role: FOLLOWER
I20260928 14:52:02.203018 8088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c3019f975c4f1982dd7a4256d275b4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36659 } }
I20260928 14:52:02.203135 8088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:52:02.203169 8088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:52:02.203212 8088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:02.204614 8088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c3019f975c4f1982dd7a4256d275b4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36659 } }
I20260928 14:52:02.204747 8088 leader_election.cc:302] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80c3019f975c4f1982dd7a4256d275b4; no voters:
I20260928 14:52:02.204919 8088 leader_election.cc:288] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:52:02.205137 8093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:52:02.205190 8088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:02.205327 8093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [term 1 LEADER]: Becoming Leader. State: Replica: 80c3019f975c4f1982dd7a4256d275b4, State: Running, Role: LEADER
I20260928 14:52:02.205524 8093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c3019f975c4f1982dd7a4256d275b4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36659 } }
I20260928 14:52:02.205943 8093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80c3019f975c4f1982dd7a4256d275b4. Latest consensus state: current_term: 1 leader_uuid: "80c3019f975c4f1982dd7a4256d275b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c3019f975c4f1982dd7a4256d275b4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36659 } } }
I20260928 14:52:02.205950 8095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80c3019f975c4f1982dd7a4256d275b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c3019f975c4f1982dd7a4256d275b4" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36659 } } }
I20260928 14:52:02.206135 8095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:02.206341 8093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:02.206525 8104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:52:02.207311 8104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:52:02.209123 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:02.202740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43475 (local address 127.18.96.190:36659)
0928 14:52:02.202988 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:02.202998 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:02.203032 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:52:02.203163 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:02.203171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:02.203235 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:02.203390 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:02.203696 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:02.205186 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:02.206506 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:02.207098 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:02.207238 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:02.208193 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:02.208197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:02.208198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:52:02.208437 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:02.208481 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:52:02.208497 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:02.208591 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:02.208863 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 14:52:02.208889 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":120,"threads_started":1}
I20260928 14:52:02.209574 8104 catalog_manager.cc:1347] Generated new cluster ID: ef12400d37174aef95e2897d6867be93
I20260928 14:52:02.209638 8104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:52:02.210279 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:52:02.216264 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:02.212049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45427 (local address 127.18.96.190:36659)
0928 14:52:02.212094 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:02.212096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:02.212235 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:52:02.212309 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:02.212310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:02.212325 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:02.212391 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:02.212553 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:02.213402 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:02.214506 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:02.214884 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:02.214976 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:02.215947 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:02.215949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:02.215949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:52:02.215956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:02.215976 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:52:02.215981 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:02.216058 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:02.216150 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 14:52:02.216168 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:52:02.216711 8104 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:52:02.217300 8104 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:52:02.226020 8104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80c3019f975c4f1982dd7a4256d275b4: Generated new TSK 0
I20260928 14:52:02.226364 8104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:52:02.234594 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 8020
W20260928 14:52:02.239470 8005 connection.cc:535] client connection to 127.18.96.190:36659 recv error: Network error: failed to read from TLS socket (remote: 127.18.96.190:36659): Connection reset by peer (error 104)
2026-09-28T14:52:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:52:02 dist-test-slave-j968 krb5kdc[8119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:52:02 dist-test-slave-j968 krb5kdc[8119](info): set up 2 sockets
Sep 28 14:52:02 dist-test-slave-j968 krb5kdc[8119](info): commencing operation
krb5kdc: starting...
W20260928 14:52:03.316982 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:52:03 dist-test-slave-j968 krb5kdc[8119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:52:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:52:03.448239 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:42721
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:52:03.530452 8132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:03.530681 8132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:03.530715 8132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:03.531937 8132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:03.531994 8132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:03.532011 8132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:03.532023 8132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:03.532037 8132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:03.533538 8132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:42721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:03.533767 8132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:03.533941 8132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:03.539592 8137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:03.539538 8138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:03.539673 8132 server_base.cc:1034] running on GCE node
W20260928 14:52:03.539534 8140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:03.540060 8132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:03.540408 8132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:03.541637 8132 hybrid_clock.cc:648] HybridClock initialized: now 1790607123541596 us; error 51 us; skew 500 ppm
Sep 28 14:52:03 dist-test-slave-j968 krb5kdc[8119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607123, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:03.544467 8132 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:52:03.545034 8132 webserver.cc:460] Webserver started at http://127.18.96.190:34207/ using document root <none> and password file <none>
I20260928 14:52:03.545372 8132 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:03.545600 8132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:03.545881 8132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:03.547585 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "e56b76108e5540a4ab66edfcce27ebb5"
format_stamp: "Formatted at 2026-09-28 14:52:03 on dist-test-slave-j968"
server_key: "a1ef197fa8240defabc5a03cbd7a2ae5"
server_key_iv: "34bfeb3dc43cf1c71223b6bae36b9e31"
server_key_version: "encryptionkey@0"
I20260928 14:52:03.548056 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "e56b76108e5540a4ab66edfcce27ebb5"
format_stamp: "Formatted at 2026-09-28 14:52:03 on dist-test-slave-j968"
server_key: "a1ef197fa8240defabc5a03cbd7a2ae5"
server_key_iv: "34bfeb3dc43cf1c71223b6bae36b9e31"
server_key_version: "encryptionkey@0"
I20260928 14:52:03.549484 8132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:52:03.550388 8147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:03.550650 8132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:52:03.550767 8132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "e56b76108e5540a4ab66edfcce27ebb5"
format_stamp: "Formatted at 2026-09-28 14:52:03 on dist-test-slave-j968"
server_key: "a1ef197fa8240defabc5a03cbd7a2ae5"
server_key_iv: "34bfeb3dc43cf1c71223b6bae36b9e31"
server_key_version: "encryptionkey@0"
I20260928 14:52:03.550850 8132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:03.554556 8132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:03.557091 8132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:03.557276 8132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:03.561719 8132 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:42721
I20260928 14:52:03.561775 8199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:42721 every 8 connection(s)
I20260928 14:52:03.562168 8132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:52:03.563299 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 8132
I20260928 14:52:03.563324 8200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:03.563441 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:52:03.563647 18818 external_mini_cluster.cc:1428] Setting key 8bc53355820e27c581ef8a16975000cf
I20260928 14:52:03.566361 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:03.564274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34315 (local address 127.18.96.190:42721)
0928 14:52:03.564445 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:03.564456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:03.564490 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:52:03.564974 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:03.564981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:03.565017 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:03.565148 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:03.565887 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:03.565890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:03.565891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:52:03.565894 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:03.565937 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:52:03.565948 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:03.566002 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:03.566098 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 14:52:03.566157 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260928 14:52:03.567384 8200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5: Bootstrap starting.
I20260928 14:52:03.567418 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:52:03.568120 8200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:03.568426 8200 log.cc:824] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:03.569429 8200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5: No bootstrap required, opened a new log
I20260928 14:52:03.570061 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:03.568976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40587 (local address 127.18.96.190:42721)
0928 14:52:03.569054 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:03.569056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:03.569147 (+ 91us) server_negotiation.cc:407] Connection header received
0928 14:52:03.569187 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:03.569187 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:03.569193 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:03.569264 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:03.569842 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:03.569843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:03.569844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:52:03.569848 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:03.569864 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:52:03.569869 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:03.569937 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:03.569972 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 14:52:03.569985 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:52:03.571702 8200 raft_consensus.cc:348] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56b76108e5540a4ab66edfcce27ebb5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42721 } }
I20260928 14:52:03.571859 8200 raft_consensus.cc:374] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:52:03.571959 8200 raft_consensus.cc:729] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56b76108e5540a4ab66edfcce27ebb5, State: Initialized, Role: FOLLOWER
I20260928 14:52:03.572113 8200 consensus_queue.cc:260] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56b76108e5540a4ab66edfcce27ebb5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42721 } }
I20260928 14:52:03.572219 8200 raft_consensus.cc:388] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:52:03.572247 8200 raft_consensus.cc:482] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:52:03.572279 8200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:03.572861 8200 raft_consensus.cc:504] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56b76108e5540a4ab66edfcce27ebb5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42721 } }
I20260928 14:52:03.572949 8200 leader_election.cc:302] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e56b76108e5540a4ab66edfcce27ebb5; no voters:
I20260928 14:52:03.573092 8200 leader_election.cc:288] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:52:03.573248 8210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:52:03.573355 8200 sys_catalog.cc:564] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:03.573438 8210 raft_consensus.cc:686] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [term 1 LEADER]: Becoming Leader. State: Replica: e56b76108e5540a4ab66edfcce27ebb5, State: Running, Role: LEADER
I20260928 14:52:03.573560 8210 consensus_queue.cc:237] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56b76108e5540a4ab66edfcce27ebb5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42721 } }
I20260928 14:52:03.574013 8211 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e56b76108e5540a4ab66edfcce27ebb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56b76108e5540a4ab66edfcce27ebb5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42721 } } }
I20260928 14:52:03.574128 8211 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:03.573997 8212 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e56b76108e5540a4ab66edfcce27ebb5. Latest consensus state: current_term: 1 leader_uuid: "e56b76108e5540a4ab66edfcce27ebb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56b76108e5540a4ab66edfcce27ebb5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 42721 } } }
I20260928 14:52:03.574205 8212 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:03.574401 8219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:52:03.575039 8219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:52:03.576759 8219 catalog_manager.cc:1347] Generated new cluster ID: 2824ef0a200b4b4496e547897ff3e540
I20260928 14:52:03.576817 8219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:52:03.585306 8219 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:52:03.585500 8219 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:52:03.588477 8219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e56b76108e5540a4ab66edfcce27ebb5: Generated new TSK 0
I20260928 14:52:03.588709 8219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:52:03.597601 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 8132
2026-09-28T14:52:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:52:03 dist-test-slave-j968 krb5kdc[8231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:52:03 dist-test-slave-j968 krb5kdc[8231](info): set up 2 sockets
Sep 28 14:52:03 dist-test-slave-j968 krb5kdc[8231](info): commencing operation
krb5kdc: starting...
W20260928 14:52:04.637044 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:52:04 dist-test-slave-j968 krb5kdc[8231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:52:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:52:04.763343 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:36861
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:36861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:52:04.846500 8244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:04.846776 8244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:04.846817 8244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:04.848078 8244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:04.848170 8244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:04.848194 8244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:04.848212 8244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:04.848229 8244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:04.849848 8244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:36861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:36861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:04.850150 8244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:04.850407 8244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:04.855587 8252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:04.855628 8250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:04.855595 8249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:04.855731 8244 server_base.cc:1034] running on GCE node
I20260928 14:52:04.856117 8244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:04.856455 8244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:04.857600 8244 hybrid_clock.cc:648] HybridClock initialized: now 1790607124857602 us; error 65 us; skew 500 ppm
Sep 28 14:52:04 dist-test-slave-j968 krb5kdc[8231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607124, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:04.864408 8244 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:52:04.864925 8244 webserver.cc:460] Webserver started at http://127.18.96.190:36795/ using document root <none> and password file <none>
I20260928 14:52:04.865242 8244 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:04.865320 8244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:04.865479 8244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:04.866653 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "9feae824121b453190fdea023747d853"
format_stamp: "Formatted at 2026-09-28 14:52:04 on dist-test-slave-j968"
server_key: "2bb362a98d919c8486d8d538e47dc763"
server_key_iv: "149e8349cc43d9010165fa08e1282937"
server_key_version: "encryptionkey@0"
I20260928 14:52:04.867038 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "9feae824121b453190fdea023747d853"
format_stamp: "Formatted at 2026-09-28 14:52:04 on dist-test-slave-j968"
server_key: "2bb362a98d919c8486d8d538e47dc763"
server_key_iv: "149e8349cc43d9010165fa08e1282937"
server_key_version: "encryptionkey@0"
I20260928 14:52:04.868657 8244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:52:04.869530 8259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:04.869757 8244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:52:04.869870 8244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "9feae824121b453190fdea023747d853"
format_stamp: "Formatted at 2026-09-28 14:52:04 on dist-test-slave-j968"
server_key: "2bb362a98d919c8486d8d538e47dc763"
server_key_iv: "149e8349cc43d9010165fa08e1282937"
server_key_version: "encryptionkey@0"
I20260928 14:52:04.869951 8244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:04.877769 8244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:04.880172 8244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:04.880345 8244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:04.884630 8244 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:36861
I20260928 14:52:04.884766 8311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:36861 every 8 connection(s)
I20260928 14:52:04.885072 8244 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:52:04.886000 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:04.888724 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 8244
I20260928 14:52:04.888744 8312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853: Bootstrap starting.
I20260928 14:52:04.888867 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:52:04.889076 18818 external_mini_cluster.cc:1428] Setting key 01994883a7bbb6aeacf2ff12ce57ed49
I20260928 14:52:04.889544 8312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:04.889961 8312 log.cc:824] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:04.890766 8312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853: No bootstrap required, opened a new log
I20260928 14:52:04.893399 8312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9feae824121b453190fdea023747d853" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36861 } }
I20260928 14:52:04.893591 8312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:52:04.893627 8312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9feae824121b453190fdea023747d853, State: Initialized, Role: FOLLOWER
I20260928 14:52:04.893765 8312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9feae824121b453190fdea023747d853" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36861 } }
I20260928 14:52:04.893891 8312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:52:04.893944 8312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:52:04.894023 8312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:04.894831 8312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9feae824121b453190fdea023747d853" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36861 } }
I20260928 14:52:04.894969 8312 leader_election.cc:302] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9feae824121b453190fdea023747d853; no voters:
I20260928 14:52:04.895160 8312 leader_election.cc:288] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:52:04.895190 8317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:52:04.895395 8317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [term 1 LEADER]: Becoming Leader. State: Replica: 9feae824121b453190fdea023747d853, State: Running, Role: LEADER
I20260928 14:52:04.895474 8312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:04.895519 8317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9feae824121b453190fdea023747d853" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36861 } }
I20260928 14:52:04.895998 8319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9feae824121b453190fdea023747d853. Latest consensus state: current_term: 1 leader_uuid: "9feae824121b453190fdea023747d853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9feae824121b453190fdea023747d853" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36861 } } }
I20260928 14:52:04.896090 8319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:04.895989 8318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9feae824121b453190fdea023747d853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9feae824121b453190fdea023747d853" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 36861 } } }
I20260928 14:52:04.896745 8318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853 [sys.catalog]: This master's current role is: LEADER
Sep 28 14:52:04 dist-test-slave-j968 krb5kdc[8231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
W20260928 14:52:04.897873 8333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:52:04.897977 8333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:52:04.898085 8322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:52:04.898314 8322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:52:04.900291 8322 catalog_manager.cc:1347] Generated new cluster ID: 6d0ef41c089c4b63899928f4a2b93e1c
I20260928 14:52:04.900349 8322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:52:04.902022 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:04.889662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53451 (local address 127.18.96.190:36861)
0928 14:52:04.889856 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:04.889864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:04.889889 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:52:04.890502 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:04.890510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:04.890561 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:04.890688 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:04.891540 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:04.892511 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:04.893807 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:04.894364 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:04.894447 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:04.897902 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:04.897906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:04.897906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:04.897919 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:04.900201 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:04.900616 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:04.900619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:04.900619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:04.900742 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:04.900970 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:04.900971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:04.900971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:04.901322 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:04.901493 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:04.901586 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 14:52:04.901757 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260928 14:52:04.903024 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:52:04.905232 8322 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:52:04.905771 8322 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:52:04.910173 8322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9feae824121b453190fdea023747d853: Generated new TSK 0
I20260928 14:52:04.910365 8322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:52:04.912814 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:04.904741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37745 (local address 127.18.96.190:36861)
0928 14:52:04.904791 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:04.904794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:04.904889 (+ 95us) server_negotiation.cc:407] Connection header received
0928 14:52:04.904999 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:04.905000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:04.905011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:04.905074 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:04.905915 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:04.906612 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:04.907766 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:04.908104 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:04.908166 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:04.909867 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:04.909868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:04.909868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:04.909875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:04.911507 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:04.911994 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:04.911995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:04.911996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:04.912053 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:04.912272 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:04.912273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:04.912273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:04.912374 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:04.912447 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:04.912641 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 14:52:04.912744 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:52:04.914831 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 8244
2026-09-28T14:52:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:52:04 dist-test-slave-j968 krb5kdc[8343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:52:04 dist-test-slave-j968 krb5kdc[8343](info): set up 2 sockets
Sep 28 14:52:04 dist-test-slave-j968 krb5kdc[8343](info): commencing operation
krb5kdc: starting...
W20260928 14:52:06.017293 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:52:06 dist-test-slave-j968 krb5kdc[8343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:52:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:52:06.143411 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34515
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:39101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:34515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:52:06.229408 8356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:06.229624 8356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:06.229646 8356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:06.230753 8356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:06.230789 8356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:06.230803 8356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:06.230815 8356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:06.230827 8356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:06.232332 8356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:39101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:34515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:34515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:06.232533 8356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:06.232710 8356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:06.237643 8364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:06.237648 8362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:06.237779 8356 server_base.cc:1034] running on GCE node
W20260928 14:52:06.237682 8361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:06.238155 8356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:06.238468 8356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:06.239631 8356 hybrid_clock.cc:648] HybridClock initialized: now 1790607126239604 us; error 39 us; skew 500 ppm
Sep 28 14:52:06 dist-test-slave-j968 krb5kdc[8343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607126, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:06.242403 8356 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:52:06.242897 8356 webserver.cc:460] Webserver started at http://127.18.96.190:46507/ using document root <none> and password file <none>
I20260928 14:52:06.243127 8356 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:06.243172 8356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:06.243355 8356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:06.244513 8356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "b44ff22e73b949a0a3997ea0d08454e5"
format_stamp: "Formatted at 2026-09-28 14:52:06 on dist-test-slave-j968"
server_key: "20ca5d9079b4da7b2299080656755681"
server_key_iv: "6a2efa42586c21071bd4d2496cf69241"
server_key_version: "encryptionkey@0"
I20260928 14:52:06.244886 8356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "b44ff22e73b949a0a3997ea0d08454e5"
format_stamp: "Formatted at 2026-09-28 14:52:06 on dist-test-slave-j968"
server_key: "20ca5d9079b4da7b2299080656755681"
server_key_iv: "6a2efa42586c21071bd4d2496cf69241"
server_key_version: "encryptionkey@0"
I20260928 14:52:06.246269 8356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:52:06.247006 8371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:06.247192 8356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:52:06.247273 8356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "b44ff22e73b949a0a3997ea0d08454e5"
format_stamp: "Formatted at 2026-09-28 14:52:06 on dist-test-slave-j968"
server_key: "20ca5d9079b4da7b2299080656755681"
server_key_iv: "6a2efa42586c21071bd4d2496cf69241"
server_key_version: "encryptionkey@0"
I20260928 14:52:06.247327 8356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:06.262219 8356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:06.264622 8356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:06.264742 8356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:06.268702 8356 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:34515
I20260928 14:52:06.268791 8423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:34515 every 8 connection(s)
I20260928 14:52:06.269131 8356 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:52:06.270047 8424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:06.272634 8424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5: Bootstrap starting.
I20260928 14:52:06.273260 8424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:06.273561 8424 log.cc:824] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:06.274299 8424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5: No bootstrap required, opened a new log
I20260928 14:52:06.276062 8424 raft_consensus.cc:348] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ff22e73b949a0a3997ea0d08454e5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34515 } }
I20260928 14:52:06.276228 8424 raft_consensus.cc:374] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:52:06.276278 8424 raft_consensus.cc:729] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44ff22e73b949a0a3997ea0d08454e5, State: Initialized, Role: FOLLOWER
I20260928 14:52:06.276470 8424 consensus_queue.cc:260] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ff22e73b949a0a3997ea0d08454e5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34515 } }
I20260928 14:52:06.276573 8424 raft_consensus.cc:388] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:52:06.276618 8424 raft_consensus.cc:482] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:52:06.276669 8424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:06.277549 8424 raft_consensus.cc:504] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ff22e73b949a0a3997ea0d08454e5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34515 } }
I20260928 14:52:06.277674 8424 leader_election.cc:302] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b44ff22e73b949a0a3997ea0d08454e5; no voters:
I20260928 14:52:06.277863 8424 leader_election.cc:288] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:52:06.277905 8427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:52:06.278070 8427 raft_consensus.cc:686] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [term 1 LEADER]: Becoming Leader. State: Replica: b44ff22e73b949a0a3997ea0d08454e5, State: Running, Role: LEADER
I20260928 14:52:06.278133 8424 sys_catalog.cc:564] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:06.278388 8427 consensus_queue.cc:237] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ff22e73b949a0a3997ea0d08454e5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34515 } }
I20260928 14:52:06.278801 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 8356
I20260928 14:52:06.278932 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:52:06.279139 18818 external_mini_cluster.cc:1428] Setting key 0ae077ba539ef05108b3222c7c5f7cab
I20260928 14:52:06.280151 8428 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b44ff22e73b949a0a3997ea0d08454e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ff22e73b949a0a3997ea0d08454e5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34515 } } }
I20260928 14:52:06.280378 8428 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:06.280578 8429 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b44ff22e73b949a0a3997ea0d08454e5. Latest consensus state: current_term: 1 leader_uuid: "b44ff22e73b949a0a3997ea0d08454e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44ff22e73b949a0a3997ea0d08454e5" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 34515 } } }
W20260928 14:52:06.280607 8443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:52:06.280792 8443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:52:06.280929 8445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:52:06.281159 8445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:52:06.280723 8429 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:06.283540 8445 catalog_manager.cc:1347] Generated new cluster ID: 6cd5e940449f4860b49d2033f1c0def6
I20260928 14:52:06.283593 8445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:52:06.286083 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:06.279704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41011 (local address 127.18.96.190:34515)
0928 14:52:06.280028 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:06.280064 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:06.280106 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:52:06.280280 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:06.280288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:06.280407 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:06.280954 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:06.281113 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:06.283024 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:06.284104 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:06.284554 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:06.284680 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:06.285426 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:06.285430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:06.285430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:52:06.285614 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:06.285645 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:52:06.285657 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:06.285740 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:06.285820 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 14:52:06.285887 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":141,"threads_started":1}
I20260928 14:52:06.288882 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:52:06.299090 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:06.292196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45975 (local address 127.18.96.190:34515)
0928 14:52:06.292270 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:06.292273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:06.296053 (+ 3780us) server_negotiation.cc:407] Connection header received
0928 14:52:06.296116 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:06.296117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:06.296130 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:06.296208 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:06.296397 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:06.296900 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:06.297708 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:06.297932 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:06.298019 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:06.298796 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:06.298797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:06.298798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:52:06.298803 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:06.298822 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:52:06.298826 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:06.298900 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:06.298960 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 14:52:06.298990 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:52:06.309989 8445 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:52:06.310796 8445 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:52:06.321924 8445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b44ff22e73b949a0a3997ea0d08454e5: Generated new TSK 0
I20260928 14:52:06.322274 8445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:52:06.325635 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 8356
2026-09-28T14:52:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:52:06 dist-test-slave-j968 krb5kdc[8455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:52:06 dist-test-slave-j968 krb5kdc[8455](info): set up 2 sockets
Sep 28 14:52:06 dist-test-slave-j968 krb5kdc[8455](info): commencing operation
krb5kdc: starting...
W20260928 14:52:07.365543 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:52:07 dist-test-slave-j968 krb5kdc[8455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:52:07.507735 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:38649
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:38649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:52:07.591559 8468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:07.591825 8468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:07.591864 8468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:07.593082 8468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:07.593170 8468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:07.593194 8468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:07.593233 8468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:07.593257 8468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:07.594835 8468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:38649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:38649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:07.595136 8468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:07.595377 8468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:52:07.600688 8468 server_base.cc:1034] running on GCE node
W20260928 14:52:07.600683 8474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:07.600831 8476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:07.600844 8473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:07.601197 8468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:07.601462 8468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:07.602626 8468 hybrid_clock.cc:648] HybridClock initialized: now 1790607127602611 us; error 29 us; skew 500 ppm
Sep 28 14:52:07 dist-test-slave-j968 krb5kdc[8455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607127, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:07.605260 8468 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:52:07.605794 8468 webserver.cc:460] Webserver started at http://127.18.96.190:35739/ using document root <none> and password file <none>
I20260928 14:52:07.606034 8468 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:07.606079 8468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:07.606220 8468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:07.607366 8468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "150de14f9ecb4eacaa94f97da0449b94"
format_stamp: "Formatted at 2026-09-28 14:52:07 on dist-test-slave-j968"
server_key: "d6a78f09c32d86efa18e03978e01e877"
server_key_iv: "bcd4c2f31b22a82b6972f9e0fcb83057"
server_key_version: "encryptionkey@0"
I20260928 14:52:07.607748 8468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "150de14f9ecb4eacaa94f97da0449b94"
format_stamp: "Formatted at 2026-09-28 14:52:07 on dist-test-slave-j968"
server_key: "d6a78f09c32d86efa18e03978e01e877"
server_key_iv: "bcd4c2f31b22a82b6972f9e0fcb83057"
server_key_version: "encryptionkey@0"
I20260928 14:52:07.609283 8468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:52:07.610069 8483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:07.610270 8468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:52:07.610373 8468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "150de14f9ecb4eacaa94f97da0449b94"
format_stamp: "Formatted at 2026-09-28 14:52:07 on dist-test-slave-j968"
server_key: "d6a78f09c32d86efa18e03978e01e877"
server_key_iv: "bcd4c2f31b22a82b6972f9e0fcb83057"
server_key_version: "encryptionkey@0"
I20260928 14:52:07.610461 8468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:07.613973 8468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:07.615909 8468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:07.616078 8468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:07.620220 8468 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:38649
I20260928 14:52:07.620271 8535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:38649 every 8 connection(s)
I20260928 14:52:07.620626 8468 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:52:07.621595 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:07.622874 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 8468
I20260928 14:52:07.622994 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:52:07.623165 18818 external_mini_cluster.cc:1428] Setting key fc8da523e907acc58ba429bda42bc25d
I20260928 14:52:07.624423 8536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94: Bootstrap starting.
I20260928 14:52:07.625147 8536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:07.625512 8536 log.cc:824] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:07.626039 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:07.623729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40565 (local address 127.18.96.190:38649)
0928 14:52:07.623975 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:07.623985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:07.624020 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:52:07.624552 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:07.624560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:07.624593 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:07.624758 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:07.625455 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:07.625460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:07.625460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:52:07.625465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:07.625508 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:52:07.625523 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:07.625615 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:07.625710 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 14:52:07.625853 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":88,"threads_started":1}
I20260928 14:52:07.626293 8536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94: No bootstrap required, opened a new log
I20260928 14:52:07.627136 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:52:07.628536 8536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150de14f9ecb4eacaa94f97da0449b94" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38649 } }
I20260928 14:52:07.628747 8536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:52:07.628829 8536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150de14f9ecb4eacaa94f97da0449b94, State: Initialized, Role: FOLLOWER
I20260928 14:52:07.628965 8536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150de14f9ecb4eacaa94f97da0449b94" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38649 } }
I20260928 14:52:07.629078 8536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:52:07.629124 8536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:52:07.629184 8536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:07.629737 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:07.628643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37521 (local address 127.18.96.190:38649)
0928 14:52:07.628686 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:07.628688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:07.628787 (+ 99us) server_negotiation.cc:407] Connection header received
0928 14:52:07.628845 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:07.628846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:07.628851 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:07.628901 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:07.629504 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:07.629506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:07.629506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:52:07.629511 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:07.629525 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:52:07.629529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:07.629579 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:07.629633 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:52:07.629646 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:52:07.629779 8536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150de14f9ecb4eacaa94f97da0449b94" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38649 } }
I20260928 14:52:07.629920 8536 leader_election.cc:302] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 150de14f9ecb4eacaa94f97da0449b94; no voters:
I20260928 14:52:07.630089 8536 leader_election.cc:288] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:52:07.630115 8546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:52:07.630275 8546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [term 1 LEADER]: Becoming Leader. State: Replica: 150de14f9ecb4eacaa94f97da0449b94, State: Running, Role: LEADER
I20260928 14:52:07.630368 8546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150de14f9ecb4eacaa94f97da0449b94" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38649 } }
I20260928 14:52:07.630399 8536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:07.630692 8546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 150de14f9ecb4eacaa94f97da0449b94. Latest consensus state: current_term: 1 leader_uuid: "150de14f9ecb4eacaa94f97da0449b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150de14f9ecb4eacaa94f97da0449b94" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38649 } } }
I20260928 14:52:07.630693 8547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "150de14f9ecb4eacaa94f97da0449b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150de14f9ecb4eacaa94f97da0449b94" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 38649 } } }
I20260928 14:52:07.630846 8546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:07.630856 8547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:07.631038 8552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:52:07.631533 8552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:52:07.633549 8552 catalog_manager.cc:1347] Generated new cluster ID: 238b65b6c2914e258a122311482edfd7
I20260928 14:52:07.633641 8552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:52:07.638280 8552 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:52:07.638442 8552 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:52:07.648308 8552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 150de14f9ecb4eacaa94f97da0449b94: Generated new TSK 0
I20260928 14:52:07.648558 8552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:52:07.651739 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 8468
2026-09-28T14:52:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1325 ms)
[----------] 18 tests from AuthTokenIssuingTest (24386 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:52:07 dist-test-slave-j968 krb5kdc[8567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:52:07 dist-test-slave-j968 krb5kdc[8567](info): set up 2 sockets
Sep 28 14:52:07 dist-test-slave-j968 krb5kdc[8567](info): commencing operation
krb5kdc: starting...
W20260928 14:52:08.692997 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:52:08 dist-test-slave-j968 krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:52:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:52:08.815373 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43989
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:43989,127.18.96.189:42461,127.18.96.188:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:52:08.896623 8580 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:52:08.896903 8580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:08.896935 8580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:08.896978 8580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:08.898151 8580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:08.898221 8580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:08.898247 8580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:08.898277 8580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:08.898308 8580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:08.899809 8580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:43989,127.18.96.189:42461,127.18.96.188:46483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:08.900115 8580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:08.900352 8580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:08.905436 8588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:08.905429 8585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:08.905429 8586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:08.905458 8580 server_base.cc:1034] running on GCE node
I20260928 14:52:08.905947 8580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:08.906241 8580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:08.907418 8580 hybrid_clock.cc:648] HybridClock initialized: now 1790607128907408 us; error 37 us; skew 500 ppm
Sep 28 14:52:08 dist-test-slave-j968 krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607128, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:08.910250 8580 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:52:08.910714 8580 webserver.cc:460] Webserver started at http://127.18.96.190:37307/ using document root <none> and password file <none>
I20260928 14:52:08.910949 8580 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:08.910995 8580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:08.911144 8580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:08.912357 8580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "284ad807f53e4a6686fb106f86ae0357"
format_stamp: "Formatted at 2026-09-28 14:52:08 on dist-test-slave-j968"
server_key: "46ac36bf1292ab178271a4463431d9b0"
server_key_iv: "9f0eaf760c22a22fe462d17cec87f2f1"
server_key_version: "encryptionkey@0"
I20260928 14:52:08.912720 8580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "284ad807f53e4a6686fb106f86ae0357"
format_stamp: "Formatted at 2026-09-28 14:52:08 on dist-test-slave-j968"
server_key: "46ac36bf1292ab178271a4463431d9b0"
server_key_iv: "9f0eaf760c22a22fe462d17cec87f2f1"
server_key_version: "encryptionkey@0"
I20260928 14:52:08.914065 8580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:52:08.914885 8595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:08.915259 8580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:52:08.915367 8580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "284ad807f53e4a6686fb106f86ae0357"
format_stamp: "Formatted at 2026-09-28 14:52:08 on dist-test-slave-j968"
server_key: "46ac36bf1292ab178271a4463431d9b0"
server_key_iv: "9f0eaf760c22a22fe462d17cec87f2f1"
server_key_version: "encryptionkey@0"
I20260928 14:52:08.915421 8580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:08.935940 8580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:08.938498 8580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:08.938658 8580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:08.942675 8580 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:43989
I20260928 14:52:08.942812 8647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:43989 every 8 connection(s)
I20260928 14:52:08.943130 8580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:52:08.944178 8648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:08.945799 8648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:08.950532 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 8580
I20260928 14:52:08.950649 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:52:08.950855 18818 external_mini_cluster.cc:1428] Setting key 6c861c9538b8813da85b8e6c1e1bf39a
Sep 28 14:52:08 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607128, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:52:08.956063 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:08.946673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.18.96.190:43989)
0928 14:52:08.946805 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:08.946809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:08.946838 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:52:08.947261 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:08.947262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:08.947279 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:08.947356 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:08.947945 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:08.948791 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:08.949609 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:08.949912 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:08.949993 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:08.952605 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:08.952615 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:08.952615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:08.952623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:08.954316 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:08.954802 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:08.954805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:08.954806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:08.954888 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:08.955119 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:08.955120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:08.955121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:08.955337 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:08.955430 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:08.955617 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 14:52:08.955837 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260928 14:52:08.956061 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:08.946495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43989 (local address 127.0.0.1:55960)
0928 14:52:08.946669 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:08.946711 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:08.947134 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:08.947386 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:08.947388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:08.947636 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:08.947810 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:08.947814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:08.948873 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:08.948874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:08.949536 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:08.949538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:08.950036 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:08.950038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:08.950103 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:08.950763 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:08.950775 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:08.952519 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:08.954424 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:08.954430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:08.954432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:08.954709 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:08.954956 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:08.954957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:08.954958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:08.955054 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:08.955407 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:08.955413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:08.955476 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 14:52:08.955585 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:52:08.955787 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260928 14:52:08.957119 8648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:08.957469 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:08.957348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42461 (local address 127.0.0.1:35180)
0928 14:52:08.957397 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:08.957440 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:52:08.957626 8599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
I20260928 14:52:08.958063 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:08.957964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42461 (local address 127.0.0.1:35186)
0928 14:52:08.958009 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:08.958040 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:52:08.958254 8648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.189:42461: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.18.96.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab.
Entry for principal kudu/127.18.96.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab.
I20260928 14:52:08.979094 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.189:42461
--webserver_interface=127.18.96.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab
--principal=kudu/127.18.96.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:43989,127.18.96.189:42461,127.18.96.188:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:52:09.007483 8648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } attempt: 1
I20260928 14:52:09.008093 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.007939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42461 (local address 127.0.0.1:35192)
0928 14:52:09.008014 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.008059 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:52:09.008515 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.008412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42461 (local address 127.0.0.1:35196)
0928 14:52:09.008462 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.008488 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 14:52:09.008610 8648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.189:42461: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
W20260928 14:52:09.061086 8655 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:52:09.061374 8655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:09.061404 8655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:09.061451 8655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:09.062647 8655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:09.062727 8655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:09.062753 8655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:09.062783 8655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:09.062814 8655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:09.064386 8655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:43989,127.18.96.189:42461,127.18.96.188:46483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab
--principal=kudu/127.18.96.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.189:42461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.96.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:09.064679 8655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:09.064913 8655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:09.070084 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:09.070065 8655 server_base.cc:1034] running on GCE node
W20260928 14:52:09.070056 8660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:09.070076 8661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:09.070601 8655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:09.070940 8655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:09.072082 8655 hybrid_clock.cc:648] HybridClock initialized: now 1790607129072073 us; error 32 us; skew 500 ppm
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:09.074623 8655 init.cc:377] Logged in from keytab as kudu/127.18.96.189@KRBTEST.COM (short username kudu)
I20260928 14:52:09.075109 8655 webserver.cc:460] Webserver started at http://127.18.96.189:39189/ using document root <none> and password file <none>
I20260928 14:52:09.075348 8655 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:09.075394 8655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:09.075562 8655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:09.076754 8648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } attempt: 2
I20260928 14:52:09.076731 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/data/instance:
uuid: "4caa51b4945f49e69d9600e33bc011ba"
format_stamp: "Formatted at 2026-09-28 14:52:09 on dist-test-slave-j968"
server_key: "b3b6435fc9e33af56b7c2aaa7d08d6ec"
server_key_iv: "f076166e703342cb303eda90b0e33099"
server_key_version: "encryptionkey@0"
I20260928 14:52:09.077131 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.077007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42461 (local address 127.0.0.1:35208)
0928 14:52:09.077064 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.077102 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:52:09.077137 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/wal/instance:
uuid: "4caa51b4945f49e69d9600e33bc011ba"
format_stamp: "Formatted at 2026-09-28 14:52:09 on dist-test-slave-j968"
server_key: "b3b6435fc9e33af56b7c2aaa7d08d6ec"
server_key_iv: "f076166e703342cb303eda90b0e33099"
server_key_version: "encryptionkey@0"
I20260928 14:52:09.077485 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.077388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42461 (local address 127.0.0.1:35222)
0928 14:52:09.077433 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.077458 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 14:52:09.077570 8648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.189:42461: Network error: Client connection negotiation failed: client connection to 127.18.96.189:42461: connect: Connection refused (error 111)
I20260928 14:52:09.078586 8655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:52:09.079331 8670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:09.079507 8655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 14:52:09.079610 8655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/wal
uuid: "4caa51b4945f49e69d9600e33bc011ba"
format_stamp: "Formatted at 2026-09-28 14:52:09 on dist-test-slave-j968"
server_key: "b3b6435fc9e33af56b7c2aaa7d08d6ec"
server_key_iv: "f076166e703342cb303eda90b0e33099"
server_key_version: "encryptionkey@0"
I20260928 14:52:09.079684 8655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:09.095722 8655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:09.097819 8655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:09.097995 8655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:09.102443 8655 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.189:42461
I20260928 14:52:09.102496 8723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.189:42461 every 8 connection(s)
I20260928 14:52:09.102893 8655 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
I20260928 14:52:09.104000 8724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:09.104334 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 8655
I20260928 14:52:09.104455 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-1/wal/instance
I20260928 14:52:09.104624 18818 external_mini_cluster.cc:1428] Setting key 999c6975e3c910df415600805722fcc6
I20260928 14:52:09.106051 8724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.96.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.188@KRBTEST.COM" created.
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:52:09.118891 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.106958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.18.96.190:43989)
0928 14:52:09.107033 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.107035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.107281 (+ 246us) server_negotiation.cc:407] Connection header received
0928 14:52:09.107842 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.107844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.107856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.107943 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.108678 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.109295 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.110335 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.110570 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.110646 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.115765 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.115767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.115767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.115775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.117335 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.117804 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.117805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.117806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.117867 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.118133 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.118134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.118134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.118263 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.118341 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.118693 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.118788 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:52:09.119113 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.106944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43989 (local address 127.0.0.1:55974)
0928 14:52:09.107134 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.107178 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:09.107725 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:09.108020 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:09.108025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:09.108363 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:09.108576 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.108581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.109396 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.109397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.110229 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.110231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.110690 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.110691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.110816 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.111394 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:09.111409 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:09.115591 (+ 4182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:09.117444 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.117455 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.117458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.117715 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.117935 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.117936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.117936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.118065 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.118333 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:09.118341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:09.118560 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 14:52:09.118673 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:52:09.118854 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
I20260928 14:52:09.119829 8724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:52:09.129016 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.120122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42461 (local address 127.0.0.1:35232)
0928 14:52:09.120315 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.120324 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:09.120358 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:09.120562 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:09.120563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:09.120694 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:09.120746 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.120748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.122086 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.122087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.122573 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.122574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.123239 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.123240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.123295 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.123715 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:09.123720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:09.126002 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:09.127982 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.127984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.127984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.128153 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.128337 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.128337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.128337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.128387 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.128737 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:09.128737 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:09.128775 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:52:09.128827 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:52:09.128938 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 14:52:09.129168 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.120197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.18.96.189:42461)
0928 14:52:09.120376 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.120382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.120409 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:52:09.120438 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.120440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.120457 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.120557 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.120822 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.121992 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.122656 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.123163 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.123247 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.126073 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.126082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.126082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.126089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.127828 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.128223 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.128226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.128226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.128282 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.128513 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.128514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.128514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.128678 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.128748 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.128851 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.129036 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260928 14:52:09.129941 8724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab.
Entry for principal kudu/127.18.96.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab.
I20260928 14:52:09.130635 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.130466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56852)
0928 14:52:09.130536 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.130590 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:52:09.131181 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.188:46483
--webserver_interface=127.18.96.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab
--principal=kudu/127.18.96.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:43989,127.18.96.189:42461,127.18.96.188:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:52:09.132004 8672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
I20260928 14:52:09.132517 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.132377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56854)
0928 14:52:09.132435 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.132472 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 14:52:09.132643 8724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:46483: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
I20260928 14:52:09.168716 8648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } attempt: 3
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607128, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:52:09.175969 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.169213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.18.96.189:42461)
0928 14:52:09.169286 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.169288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.169300 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:52:09.169333 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.169334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.169347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.169421 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.169776 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.170280 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.171050 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.171277 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.171366 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.173244 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.173245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.173246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.173254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.174726 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.175107 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.175109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.175109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.175171 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.175370 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.175371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.175371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.175498 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.175583 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.175765 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.175845 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:52:09.175969 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.169141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42461 (local address 127.0.0.1:35246)
0928 14:52:09.169188 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.169201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:09.169277 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:09.169427 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:09.169428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:09.169617 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:09.169681 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.169683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.170376 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.170378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.170928 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.170930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.171366 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.171367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.171451 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.171956 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:09.171961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:09.173148 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:09.174829 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.174830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.174830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.175014 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.175238 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.175239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.175240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.175305 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.175593 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:09.175594 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:09.175643 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 14:52:09.175723 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:52:09.175840 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 14:52:09.176339 8648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:09.176717 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.176606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56856)
0928 14:52:09.176657 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.176687 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:52:09.177202 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.177055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56868)
0928 14:52:09.177135 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.177167 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 14:52:09.177333 8648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:46483: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
I20260928 14:52:09.181910 8724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } attempt: 1
I20260928 14:52:09.182392 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.182252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56870)
0928 14:52:09.182311 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.182348 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 14:52:09.182758 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.182652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56876)
0928 14:52:09.182698 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.182728 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:52:09.182847 8724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:46483: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
I20260928 14:52:09.208534 8648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } attempt: 1
I20260928 14:52:09.209043 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.208890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56884)
0928 14:52:09.208963 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.209003 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:52:09.209497 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.209367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56892)
0928 14:52:09.209399 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.209475 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 14:52:09.209589 8648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:46483: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
W20260928 14:52:09.216327 8731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:52:09.216609 8731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:09.216661 8731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:09.216735 8731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:09.218098 8731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:09.218181 8731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:09.218205 8731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:09.218225 8731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:09.218257 8731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:09.220158 8731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:43989,127.18.96.189:42461,127.18.96.188:46483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab
--principal=kudu/127.18.96.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.188:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.96.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:09.220504 8731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:09.220775 8731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:09.226240 8739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:09.226277 8731 server_base.cc:1034] running on GCE node
W20260928 14:52:09.226261 8737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:09.226233 8736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:09.226735 8731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:09.227074 8731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:09.228233 8731 hybrid_clock.cc:648] HybridClock initialized: now 1790607129228212 us; error 36 us; skew 500 ppm
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:09.230939 8731 init.cc:377] Logged in from keytab as kudu/127.18.96.188@KRBTEST.COM (short username kudu)
I20260928 14:52:09.231441 8731 webserver.cc:460] Webserver started at http://127.18.96.188:33759/ using document root <none> and password file <none>
I20260928 14:52:09.231684 8731 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:09.231753 8731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:09.231957 8731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:09.233093 8731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/data/instance:
uuid: "b72b8c04b5e44db5a6cd1429d692cd76"
format_stamp: "Formatted at 2026-09-28 14:52:09 on dist-test-slave-j968"
server_key: "48a2efbad17aebb07bd527002c0f1c6e"
server_key_iv: "7a41c8f33a0b2a2ab4d38ad84ba5b9a3"
server_key_version: "encryptionkey@0"
I20260928 14:52:09.233467 8731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/wal/instance:
uuid: "b72b8c04b5e44db5a6cd1429d692cd76"
format_stamp: "Formatted at 2026-09-28 14:52:09 on dist-test-slave-j968"
server_key: "48a2efbad17aebb07bd527002c0f1c6e"
server_key_iv: "7a41c8f33a0b2a2ab4d38ad84ba5b9a3"
server_key_version: "encryptionkey@0"
I20260928 14:52:09.234896 8731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 14:52:09.235733 8746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:09.235963 8731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:52:09.236068 8731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/wal
uuid: "b72b8c04b5e44db5a6cd1429d692cd76"
format_stamp: "Formatted at 2026-09-28 14:52:09 on dist-test-slave-j968"
server_key: "48a2efbad17aebb07bd527002c0f1c6e"
server_key_iv: "7a41c8f33a0b2a2ab4d38ad84ba5b9a3"
server_key_version: "encryptionkey@0"
I20260928 14:52:09.236156 8731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:09.251004 8724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } attempt: 2
I20260928 14:52:09.251475 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.251342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56898)
0928 14:52:09.251406 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.251445 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:52:09.251828 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.251733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56904)
0928 14:52:09.251779 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.251803 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:52:09.251969 8724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:46483: Network error: Client connection negotiation failed: client connection to 127.18.96.188:46483: connect: Connection refused (error 111)
I20260928 14:52:09.264753 8731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:09.267242 8731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:09.267410 8731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:09.271667 8731 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.188:46483
I20260928 14:52:09.271703 8799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.188:46483 every 8 connection(s)
I20260928 14:52:09.272105 8731 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
I20260928 14:52:09.273128 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:09.274714 8800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:09.277192 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 8731
I20260928 14:52:09.277354 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606857360589-18818-0/minicluster-data/master-2/wal/instance
I20260928 14:52:09.277524 18818 external_mini_cluster.cc:1428] Setting key 6288c590fb50c19a51ff0d2a06253644
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:52:09.284336 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.275445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.18.96.190:43989)
0928 14:52:09.275516 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.275518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.275766 (+ 248us) server_negotiation.cc:407] Connection header received
0928 14:52:09.276302 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.276303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.276313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.276366 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.276989 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.277531 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.278829 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.279030 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.279115 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.281364 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.281366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.281366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.281375 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.282898 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.283295 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.283296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.283297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.283352 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.283622 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.283624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.283624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.283761 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.283810 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.284203 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.284271 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:52:09.284631 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.275444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:43989 (local address 127.0.0.1:55986)
0928 14:52:09.275640 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.275676 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:09.276209 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:09.276443 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:09.276446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:09.276686 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:09.276910 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.276914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.277620 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.277621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.278732 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.278734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.279150 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.279151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.279268 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.279778 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:09.279786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:09.281276 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:09.282978 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.282985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.282986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.283198 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.283424 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.283424 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.283425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.283527 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.283825 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:09.283830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:09.284029 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 14:52:09.284164 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 14:52:09.284353 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260928 14:52:09.284713 8648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } attempt: 2
I20260928 14:52:09.285192 8800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:09.286796 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.277929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.18.96.188:46483)
0928 14:52:09.278111 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.278116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.278167 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:52:09.278214 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.278217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.278236 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.278319 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.278885 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.279732 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.280509 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.280918 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.281001 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.283700 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.283711 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.283711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.283717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.285548 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.285950 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.285953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.285953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.286013 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.286232 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.286233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.286233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.286385 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.286533 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.286622 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.286697 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":119,"threads_started":1}
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607128, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:52:09.293092 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.285443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:42461 (local address 127.0.0.1:35256)
0928 14:52:09.285531 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.285541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:09.285574 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:09.285697 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:09.285698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:09.285829 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:09.285893 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.285895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.286746 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.286747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.287444 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.287447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.287991 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.287991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.288057 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.288597 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:09.288602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:09.289836 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:09.291911 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.291912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.291913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.292210 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.292434 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.292435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.292435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.292488 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.292785 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:09.292785 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:09.292828 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:52:09.292899 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:52:09.292990 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 14:52:09.293390 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.285540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.18.96.189:42461)
0928 14:52:09.285588 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.285590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.285598 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:09.285626 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.285626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.285635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.285694 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.285993 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.286647 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.287527 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.287849 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.287946 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.289924 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.289926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.289926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.289934 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.291795 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.292304 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.292305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.292305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.292360 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.292558 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.292559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.292560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.292686 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.292732 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.292961 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.293102 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 14:52:09.293780 8800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:09.295905 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.284946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56924)
0928 14:52:09.285028 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.285038 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:09.285076 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:09.285455 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:09.285456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:09.285614 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:09.285658 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.285660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.286534 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.286535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.287763 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.287766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.288360 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.288361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.288421 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.288817 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:09.288822 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:09.290563 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:09.294171 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.294172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.294173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.294951 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.295233 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.295234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.295235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.295323 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.295607 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:09.295608 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:09.295649 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:52:09.295698 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:52:09.295784 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 14:52:09.296154 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.287716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.18.96.190:43989)
0928 14:52:09.287754 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.287755 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.287763 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:09.287822 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.287823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.287832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.287906 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.288495 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.289123 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.290099 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.290300 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.290384 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.293024 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.293026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.293026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.293032 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.294749 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.295138 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.295139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.295140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.295195 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.295418 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.295419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.295419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.295544 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.295682 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.295969 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.296069 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:52:09.296339 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.285154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.18.96.188:46483)
0928 14:52:09.285319 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.285322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.285330 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:09.285349 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.285350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.285361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.285436 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.285748 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.286449 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.287980 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.288284 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.288359 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.290639 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.290640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.290641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.290647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.294077 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.295024 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.295025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.295025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.295116 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.295423 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.295424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.295424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.295533 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.296080 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.296118 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.296211 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260928 14:52:09.298061 8648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357: Bootstrap starting.
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:52:09.298872 8648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:09.299197 8648 log.cc:824] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:09.300302 8648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357: No bootstrap required, opened a new log
I20260928 14:52:09.301867 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.294078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.18.96.188:46483)
0928 14:52:09.295028 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.295030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.295037 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:52:09.295053 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.295054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.295066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.295115 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.295402 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.296125 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.296981 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.297195 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.297270 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.299050 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.299051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.299052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.299057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.300718 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.301075 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.301076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.301076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.301125 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.301342 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.301343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.301343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.301478 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.301554 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.301678 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.301763 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260928 14:52:09.302152 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.294021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56932)
0928 14:52:09.294063 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.294072 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:09.294104 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:09.295150 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:09.295152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:09.295297 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:09.295329 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.295331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.296197 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.296198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.296913 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.296916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.297266 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.297267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.297315 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.297711 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:09.297714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:09.298959 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:09.300810 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.300811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.300811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.300992 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.301213 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.301214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.301214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.301267 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.301546 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:09.301547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:09.301585 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:52:09.301653 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:52:09.301911 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":161}
I20260928 14:52:09.303846 8800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76: Bootstrap starting.
I20260928 14:52:09.304102 8648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } }
I20260928 14:52:09.304252 8648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284ad807f53e4a6686fb106f86ae0357, State: Initialized, Role: FOLLOWER
I20260928 14:52:09.304352 8648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } }
I20260928 14:52:09.304486 8800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76: Neither blocks nor log segments found. Creating new log.
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:52:09.304726 8648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:09.304677 8807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.304778 8807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:09.304843 8800 log.cc:824] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:09.305709 8800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76: No bootstrap required, opened a new log
W20260928 14:52:09.306922 8819 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:52:09.306969 8819 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:52:09.307977 8800 raft_consensus.cc:348] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } }
I20260928 14:52:09.308117 8800 raft_consensus.cc:729] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72b8c04b5e44db5a6cd1429d692cd76, State: Initialized, Role: FOLLOWER
I20260928 14:52:09.308190 8800 consensus_queue.cc:260] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } }
I20260928 14:52:09.308512 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.296890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.18.96.189:42461)
0928 14:52:09.299526 (+ 2636us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.299528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.299537 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:52:09.299560 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.299561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.299578 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.299658 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.300341 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.301032 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.302009 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.302317 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.302358 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.305050 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.305052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.305052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.305060 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.306893 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.307335 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.307336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.307337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.307399 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.307599 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.307600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.307600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.307733 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.307908 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.308309 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.308393 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2612}
I20260928 14:52:09.308804 8800 sys_catalog.cc:564] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:09.307699 8580 master_runner.cc:186] Error getting master registration for 127.18.96.189:42461: OK, instance_id { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" instance_seqno: 1790607129101079 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 14:52:09.311139 8832 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:52:09.311241 8832 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:52:09.311652 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:52:09.312737 8580 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:52:09.312608 8622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "284ad807f53e4a6686fb106f86ae0357"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55996
I20260928 14:52:09.312850 8622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:52:09.312894 8622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:09.313834 8622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } }
I20260928 14:52:09.314050 8622 leader_election.cc:288] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [CANDIDATE]: Term 1 election: Requested vote from peers 4caa51b4945f49e69d9600e33bc011ba (127.18.96.189:42461), b72b8c04b5e44db5a6cd1429d692cd76 (127.18.96.188:46483)
I20260928 14:52:09.314322 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "284ad807f53e4a6686fb106f86ae0357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b72b8c04b5e44db5a6cd1429d692cd76"
I20260928 14:52:09.314421 8773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:09.312608 8731 master_runner.cc:186] Error getting master registration for 127.18.96.189:42461: OK, instance_id { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" instance_seqno: 1790607129101079 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:52:09.315155 8731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:52:09.315263 8773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 284ad807f53e4a6686fb106f86ae0357 in term 1.
I20260928 14:52:09.315478 8597 leader_election.cc:302] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284ad807f53e4a6686fb106f86ae0357, b72b8c04b5e44db5a6cd1429d692cd76; no voters:
I20260928 14:52:09.315611 8807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:52:09.315778 8807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [term 1 LEADER]: Becoming Leader. State: Replica: 284ad807f53e4a6686fb106f86ae0357, State: Running, Role: LEADER
I20260928 14:52:09.315883 8807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } }
I20260928 14:52:09.316352 8807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 284ad807f53e4a6686fb106f86ae0357. Latest consensus state: current_term: 1 leader_uuid: "284ad807f53e4a6686fb106f86ae0357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.316450 8807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:09.318068 8837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:52:09.318364 8837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:52:09.319971 8821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.320075 8821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:09.320132 8773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [term 1 FOLLOWER]: Refusing update from remote peer 284ad807f53e4a6686fb106f86ae0357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:52:09.320583 8807 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:52:09.322023 8821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 284ad807f53e4a6686fb106f86ae0357. Latest consensus state: current_term: 1 leader_uuid: "284ad807f53e4a6686fb106f86ae0357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.322206 8821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:09.327320 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.317253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.18.96.188:46483)
0928 14:52:09.317307 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.317310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.317622 (+ 312us) server_negotiation.cc:407] Connection header received
0928 14:52:09.317787 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.317788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.317812 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.317844 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.319316 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.320030 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.321598 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.321888 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.321937 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.323846 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.323847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.323848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.323856 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.325732 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.326126 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.326127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.326128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.326205 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.326721 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.326722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.326722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.326857 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.326940 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.327140 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.327232 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:52:09.333067 8834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "284ad807f53e4a6686fb106f86ae0357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.333941 8834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:09.333814 8821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "284ad807f53e4a6686fb106f86ae0357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.334012 8821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:09.333189 8843 mvcc.cc:205] Tried to move back new op lower bound from 7334326801692635136 to 7334326801678712832. Current Snapshot: MvccSnapshot[applied={T|T < 7334326801692635136}]
I20260928 14:52:09.334132 8844 mvcc.cc:205] Tried to move back new op lower bound from 7334326801692635136 to 7334326801678712832. Current Snapshot: MvccSnapshot[applied={T|T < 7334326801692635136}]
I20260928 14:52:09.337903 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.317111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.18.96.190:43989)
0928 14:52:09.317160 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.317163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.324041 (+ 6878us) server_negotiation.cc:407] Connection header received
0928 14:52:09.324148 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.324149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.324162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.324468 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.326579 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.327306 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.329517 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.329802 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.329845 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.334371 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.334373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.334373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.334382 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.336474 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.336900 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.336902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.336902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.336958 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.337203 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.337204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.337204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.337363 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.337452 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.337685 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.337781 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:52:09.339781 8837 catalog_manager.cc:1347] Generated new cluster ID: d8a245523ac142a0bc042c9ed018c06e
I20260928 14:52:09.339897 8837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:52:09.342073 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.324108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.18.96.189:42461)
0928 14:52:09.324155 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.324157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.328418 (+ 4261us) server_negotiation.cc:407] Connection header received
0928 14:52:09.328492 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.328494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.328507 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.328562 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.329651 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.330357 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.332106 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.332905 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.332958 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.336612 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.336614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.336615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.336624 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.338447 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.341165 (+ 2718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.341167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.341168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.341237 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.341451 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.341452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.341452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.341574 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.341686 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.341842 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.341929 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 14:52:09.343119 8724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } attempt: 3
Sep 28 14:52:09 dist-test-slave-j968 krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607129, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:52:09.352196 8837 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:52:09.352368 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.344062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:46483 (local address 127.0.0.1:56956)
0928 14:52:09.344115 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:09.344129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:09.344189 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:09.344495 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:09.344497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:09.344675 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:09.344727 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.344730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.345662 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.345663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.346494 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:09.346497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.347036 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.347037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:09.347106 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.347609 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:09.347615 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:09.349189 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:09.351097 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.351098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.351098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.351314 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.351568 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:09.351569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:09.351569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:09.351640 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:09.352022 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:09.352023 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:09.352082 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 14:52:09.352144 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 14:52:09.352257 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 14:52:09.352685 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.344312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.18.96.188:46483)
0928 14:52:09.344352 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.344354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.344362 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:09.344386 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.344386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.344396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.344445 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.344816 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.345553 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.346598 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.346905 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.346967 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.349306 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:09.349308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:09.349308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:09.349315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:09.351007 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.351414 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.351415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.351415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.351487 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:09.351726 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:09.351727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:09.351727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:09.351916 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:09.351978 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.352530 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.352603 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:52:09.355037 8724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba: Bootstrap starting.
I20260928 14:52:09.355830 8724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:09.356195 8724 log.cc:824] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:09.356936 8837 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:52:09.357271 8724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba: No bootstrap required, opened a new log
I20260928 14:52:09.359361 8724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } }
I20260928 14:52:09.359503 8724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4caa51b4945f49e69d9600e33bc011ba, State: Initialized, Role: FOLLOWER
I20260928 14:52:09.359579 8724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } }
I20260928 14:52:09.360284 8698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "284ad807f53e4a6686fb106f86ae0357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4caa51b4945f49e69d9600e33bc011ba"
I20260928 14:52:09.360899 8697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:09.361866 8697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [term 1 FOLLOWER]: Refusing update from remote peer 284ad807f53e4a6686fb106f86ae0357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:52:09.362227 8834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:52:09.362936 8698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 284ad807f53e4a6686fb106f86ae0357 in term 1.
I20260928 14:52:09.363564 8724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:09.364482 8837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357: Generated new TSK 0
I20260928 14:52:09.364665 8837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:52:09.365952 8854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "284ad807f53e4a6686fb106f86ae0357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.366086 8854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:09.366144 8854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 284ad807f53e4a6686fb106f86ae0357. Latest consensus state: current_term: 1 leader_uuid: "284ad807f53e4a6686fb106f86ae0357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.366183 8854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:09.366225 8854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "284ad807f53e4a6686fb106f86ae0357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.366261 8854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:09.366850 8852 mvcc.cc:205] Tried to move back new op lower bound from 7334326801859719168 to 7334326801678712832. Current Snapshot: MvccSnapshot[applied={T|T < 7334326801692635136}]
I20260928 14:52:09.369585 8865 catalog_manager.cc:1259] Loaded cluster ID: d8a245523ac142a0bc042c9ed018c06e
I20260928 14:52:09.369702 8865 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba: loading cluster ID for follower catalog manager: success
I20260928 14:52:09.370827 8865 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba: acquiring CA information for follower catalog manager: success
I20260928 14:52:09.371304 8865 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4caa51b4945f49e69d9600e33bc011ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 14:52:09.388877 8729 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.387436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.18.96.189:42461)
0928 14:52:09.387488 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.387490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.388174 (+ 684us) server_negotiation.cc:407] Connection header received
0928 14:52:09.388295 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.388296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.388309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.388340 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.388806 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35284: BlockingRecv error: recv got EOF from 127.0.0.1:35284 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 14:52:09.389818 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.387889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.18.96.188:46483)
0928 14:52:09.387931 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.387932 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.389311 (+ 1379us) server_negotiation.cc:407] Connection header received
0928 14:52:09.389410 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.389411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.389421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.389450 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.389756 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56958: BlockingRecv error: recv got EOF from 127.0.0.1:56958 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 14:52:09.390949 8651 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.390299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.18.96.190:43989)
0928 14:52:09.390339 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.390340 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.390451 (+ 111us) server_negotiation.cc:407] Connection header received
0928 14:52:09.390546 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.390547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.390557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.390586 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.390888 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56024: BlockingRecv error: recv got EOF from 127.0.0.1:56024 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:52:09.396135 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.392596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.18.96.189:42461)
0928 14:52:09.392632 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.392634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.392859 (+ 225us) server_negotiation.cc:407] Connection header received
0928 14:52:09.392969 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.392970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.393015 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.393062 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.393285 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.394066 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.395223 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.395476 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.395517 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.395744 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.395893 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.395941 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.396043 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.396063 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 14:52:09.399590 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.398797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.18.96.188:46483)
0928 14:52:09.398835 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.398837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.399031 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:52:09.399140 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.399141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.399161 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.399202 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.399538 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56970: BlockingRecv error: recv got EOF from 127.0.0.1:56970 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:52:09.405332 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.401942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.18.96.189:42461)
0928 14:52:09.401998 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.402000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.402268 (+ 268us) server_negotiation.cc:407] Connection header received
0928 14:52:09.402362 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.402363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.402373 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.402401 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.402567 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.403313 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.404441 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.404759 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.404814 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.405053 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.405090 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.405117 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.405235 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.405256 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:52:09.409143 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.408234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.18.96.188:46483)
0928 14:52:09.408281 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.408283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.408534 (+ 251us) server_negotiation.cc:407] Connection header received
0928 14:52:09.408661 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.408662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.408676 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.408707 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.409063 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56972: BlockingRecv error: recv got EOF from 127.0.0.1:56972 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 14:52:09.416359 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.412781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.18.96.189:42461)
0928 14:52:09.412825 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.412828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.413103 (+ 275us) server_negotiation.cc:407] Connection header received
0928 14:52:09.413213 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.413215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.413226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.413255 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.413431 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.414194 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.415419 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.415713 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.415756 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.416052 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.416091 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.416119 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.416239 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.416261 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 14:52:09.419771 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.418856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.18.96.188:46483)
0928 14:52:09.418896 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.418898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.419106 (+ 208us) server_negotiation.cc:407] Connection header received
0928 14:52:09.419218 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.419219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.419231 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.419260 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.419712 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56980: BlockingRecv error: recv got EOF from 127.0.0.1:56980 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 14:52:09.428476 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.425132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.18.96.189:42461)
0928 14:52:09.425175 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.425178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.425465 (+ 287us) server_negotiation.cc:407] Connection header received
0928 14:52:09.425564 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.425565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.425575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.425604 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.425787 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.426487 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.427586 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.427913 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.427966 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.428179 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.428231 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.428262 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.428375 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.428397 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:52:09.432097 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.431131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.18.96.188:46483)
0928 14:52:09.431176 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.431178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.431402 (+ 224us) server_negotiation.cc:407] Connection header received
0928 14:52:09.431502 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.431503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.431515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.431545 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.432043 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56994: BlockingRecv error: recv got EOF from 127.0.0.1:56994 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:52:09.445046 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.441518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.18.96.189:42461)
0928 14:52:09.441564 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.441566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.441859 (+ 293us) server_negotiation.cc:407] Connection header received
0928 14:52:09.441985 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.441987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.441999 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.442029 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.442245 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.442973 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.444155 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.444460 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.444510 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.444727 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.444765 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.444796 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.444941 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.444963 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:52:09.448801 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.447965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.18.96.188:46483)
0928 14:52:09.448006 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.448008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.448232 (+ 224us) server_negotiation.cc:407] Connection header received
0928 14:52:09.448336 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.448337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.448349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.448378 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.448744 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57004: BlockingRecv error: recv got EOF from 127.0.0.1:57004 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:52:09.469811 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.466288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.18.96.189:42461)
0928 14:52:09.466339 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.466342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.466723 (+ 381us) server_negotiation.cc:407] Connection header received
0928 14:52:09.466824 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.466826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.466837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.466865 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.467075 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.467808 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.468991 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.469284 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.469331 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.469540 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.469576 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.469606 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.469710 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.469730 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 14:52:09.473352 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.472478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.18.96.188:46483)
0928 14:52:09.472524 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.472526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.472745 (+ 219us) server_negotiation.cc:407] Connection header received
0928 14:52:09.472848 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.472849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.472863 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.472893 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.473291 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57016: BlockingRecv error: recv got EOF from 127.0.0.1:57016 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:52:09.509768 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.506867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.18.96.189:42461)
0928 14:52:09.506920 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.506922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.507121 (+ 199us) server_negotiation.cc:407] Connection header received
0928 14:52:09.507198 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.507199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.507211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.507284 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.507410 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.507993 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.508963 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.509180 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.509288 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.509449 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.509487 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.509537 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.509623 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.509643 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 14:52:09.512255 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.511652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.18.96.188:46483)
0928 14:52:09.511698 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.511700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.511717 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:52:09.511793 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.511794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.511805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.511851 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.512197 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57022: BlockingRecv error: recv got EOF from 127.0.0.1:57022 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:52:09.580281 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.577536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.18.96.189:42461)
0928 14:52:09.577643 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.577645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.577753 (+ 108us) server_negotiation.cc:407] Connection header received
0928 14:52:09.577808 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.577809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.577818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.577871 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.578029 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.578515 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.579496 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.579675 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.579736 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.579953 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.579982 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.580022 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.580134 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.580155 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 14:52:09.583093 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.582367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.18.96.188:46483)
0928 14:52:09.582451 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.582454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.582582 (+ 128us) server_negotiation.cc:407] Connection header received
0928 14:52:09.582615 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.582617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.582632 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.582684 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.583023 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57026: BlockingRecv error: recv got EOF from 127.0.0.1:57026 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:52:09.714663 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.712386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.18.96.189:42461)
0928 14:52:09.712461 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.712463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.712537 (+ 74us) server_negotiation.cc:407] Connection header received
0928 14:52:09.712609 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.712610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.712620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.712664 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.712798 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.713281 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.714030 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.714209 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.714264 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.714436 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.714470 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.714524 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.714578 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.714597 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:52:09.717393 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.716758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.18.96.188:46483)
0928 14:52:09.716828 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.716830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.716839 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:52:09.716936 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.716937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.716949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.717007 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.717326 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57042: BlockingRecv error: recv got EOF from 127.0.0.1:57042 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:52:09.851114 8807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "284ad807f53e4a6686fb106f86ae0357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ad807f53e4a6686fb106f86ae0357" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 43989 } } peers { permanent_uuid: "4caa51b4945f49e69d9600e33bc011ba" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 42461 } } peers { permanent_uuid: "b72b8c04b5e44db5a6cd1429d692cd76" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 46483 } } }
I20260928 14:52:09.851290 8807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284ad807f53e4a6686fb106f86ae0357 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:09.977394 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:09.974765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.18.96.189:42461)
0928 14:52:09.974847 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.974849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.974876 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:52:09.974944 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.974945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.974955 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.975077 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:09.975288 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.975829 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.976656 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:09.976856 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:09.976915 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:09.977062 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:09.977092 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:09.977135 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:09.977229 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 14:52:09.977246 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 14:52:09.979933 8804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:09.979312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.18.96.188:46483)
0928 14:52:09.979379 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:09.979381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:09.979403 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:52:09.979450 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:09.979451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:09.979462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:09.979496 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:09.979836 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57048: BlockingRecv error: recv got EOF from 127.0.0.1:57048 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:52:10.311837 8832 catalog_manager.cc:1259] Loaded cluster ID: d8a245523ac142a0bc042c9ed018c06e
I20260928 14:52:10.311972 8832 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76: loading cluster ID for follower catalog manager: success
I20260928 14:52:10.312980 8832 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76: acquiring CA information for follower catalog manager: success
I20260928 14:52:10.313328 8832 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b72b8c04b5e44db5a6cd1429d692cd76: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:52:10.495990 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:10.493384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.18.96.189:42461)
0928 14:52:10.493524 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:10.493526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:10.493543 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:52:10.493571 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:10.493571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:10.493588 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:10.493662 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:10.493866 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.494416 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.495259 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.495469 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.495534 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:10.495676 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:10.495707 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:10.495749 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:10.495802 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 14:52:10.495821 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260928 14:52:10.500185 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:10.497710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.18.96.188:46483)
0928 14:52:10.497828 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:10.497830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:10.497838 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:10.497888 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:10.497889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:10.497942 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:10.498010 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:10.498128 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.498596 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.499331 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.499559 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.499625 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:10.499760 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:10.499927 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:10.500006 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:10.500027 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 14:52:10.500052 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260928 14:52:10.504959 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:10.501893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.18.96.189:42461)
0928 14:52:10.501945 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:10.501948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:10.502207 (+ 259us) server_negotiation.cc:407] Connection header received
0928 14:52:10.502239 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:10.502240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:10.502254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:10.502309 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:10.502430 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.503045 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.504141 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.504428 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.504491 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:10.504692 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:10.504730 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:10.504783 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:10.504855 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 14:52:10.504883 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:52:10.505177 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:10.501947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.18.96.188:46483)
0928 14:52:10.501999 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:10.502001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:10.502434 (+ 433us) server_negotiation.cc:407] Connection header received
0928 14:52:10.502519 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:10.502520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:10.502531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:10.502580 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:10.502751 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.503449 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.504306 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.504596 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.504653 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:10.504853 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:10.504885 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:10.504934 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:10.505076 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 14:52:10.505113 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:52:10.505481 8998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:10.502123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.18.96.190:43989)
0928 14:52:10.502227 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:10.502229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:10.502237 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:10.502282 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:10.502282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:10.502335 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:10.502419 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:10.502583 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.503367 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.504342 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:10.504686 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:10.504749 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:10.504990 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:10.505112 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:10.505194 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:10.505354 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 14:52:10.505386 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
I20260928 14:52:10.516345 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 8580
I20260928 14:52:10.521951 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 8655
I20260928 14:52:10.526829 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 8731
2026-09-28T14:52:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2877 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:52:10 dist-test-slave-j968 krb5kdc[9001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:52:10 dist-test-slave-j968 krb5kdc[9001](info): set up 2 sockets
Sep 28 14:52:10 dist-test-slave-j968 krb5kdc[9001](info): commencing operation
krb5kdc: starting...
W20260928 14:52:11.573292 18818 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/test-user.keytab.
Sep 28 14:52:11 dist-test-slave-j968 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:52:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
Entry for principal kudu/127.18.96.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab.
I20260928 14:52:11.700942 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:41071
--webserver_interface=127.18.96.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:41071,127.18.96.189:38743,127.18.96.188:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:52:11.791795 9014 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:52:11.792133 9014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:11.792171 9014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:11.792219 9014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:11.793545 9014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:11.793632 9014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:11.793661 9014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:11.793697 9014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:11.793723 9014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:11.795454 9014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:41071,127.18.96.189:38743,127.18.96.188:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.190.keytab
--principal=kudu/127.18.96.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.190:41071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:11.795778 9014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:11.796070 9014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:11.801635 9019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:11.801808 9014 server_base.cc:1034] running on GCE node
W20260928 14:52:11.801560 9022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:11.801582 9020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:11.802217 9014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:11.802589 9014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:11.803771 9014 hybrid_clock.cc:648] HybridClock initialized: now 1790607131803749 us; error 38 us; skew 500 ppm
Sep 28 14:52:11 dist-test-slave-j968 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:11.806527 9014 init.cc:377] Logged in from keytab as kudu/127.18.96.190@KRBTEST.COM (short username kudu)
I20260928 14:52:11.807013 9014 webserver.cc:460] Webserver started at http://127.18.96.190:42909/ using document root <none> and password file <none>
I20260928 14:52:11.807250 9014 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:11.807319 9014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:11.807476 9014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:11.808647 9014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/data/instance:
uuid: "bf710f93892240bd87801b6024c62205"
format_stamp: "Formatted at 2026-09-28 14:52:11 on dist-test-slave-j968"
server_key: "b14ebf5805d1d294ddbf319c344dd797"
server_key_iv: "3f325b7e832038a559091421511c5cd1"
server_key_version: "encryptionkey@0"
I20260928 14:52:11.809021 9014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/wal/instance:
uuid: "bf710f93892240bd87801b6024c62205"
format_stamp: "Formatted at 2026-09-28 14:52:11 on dist-test-slave-j968"
server_key: "b14ebf5805d1d294ddbf319c344dd797"
server_key_iv: "3f325b7e832038a559091421511c5cd1"
server_key_version: "encryptionkey@0"
I20260928 14:52:11.810397 9014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:52:11.811177 9029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:11.811353 9014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 14:52:11.811451 9014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/wal
uuid: "bf710f93892240bd87801b6024c62205"
format_stamp: "Formatted at 2026-09-28 14:52:11 on dist-test-slave-j968"
server_key: "b14ebf5805d1d294ddbf319c344dd797"
server_key_iv: "3f325b7e832038a559091421511c5cd1"
server_key_version: "encryptionkey@0"
I20260928 14:52:11.811532 9014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:11.822028 9014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:11.824383 9014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:11.824581 9014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:11.829101 9014 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.190:41071
I20260928 14:52:11.829159 9081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.190:41071 every 8 connection(s)
I20260928 14:52:11.829581 9014 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/data/info.pb
I20260928 14:52:11.830624 9082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:11.832247 9082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:11.836658 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 9014
I20260928 14:52:11.836759 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-0/wal/instance
I20260928 14:52:11.836980 18818 external_mini_cluster.cc:1428] Setting key 9b6495722ffbf8bef7951bb61e67fdbd
Sep 28 14:52:11 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:52:11.843108 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:11.833101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.18.96.190:41071)
0928 14:52:11.833230 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:11.833237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:11.833271 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:52:11.833789 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:11.833791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:11.833808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:11.833887 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:11.834418 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:11.835248 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:11.836192 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:11.836503 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:11.836585 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:11.839166 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:11.839174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:11.839174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:11.839184 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:11.841256 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:11.841749 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:11.841753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:11.841754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:11.841910 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:11.842268 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:11.842269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:11.842269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:11.842500 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:11.842570 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:11.842831 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 14:52:11.842938 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260928 14:52:11.843250 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:11.832946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:41071 (local address 127.0.0.1:60514)
0928 14:52:11.833116 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:11.833165 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:11.833644 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:11.833915 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:11.833918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:11.834169 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:11.834335 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:11.834339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:11.835341 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:11.835342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:11.836120 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:11.836122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:11.836654 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:11.836656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:11.836743 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:11.837387 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:11.837397 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:11.839060 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:11.841380 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:11.841389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:11.841392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:11.841655 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:11.841995 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:11.841996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:11.841996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:11.842175 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:11.842579 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:11.842587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:11.842676 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 14:52:11.842795 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 14:52:11.843062 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":64,"thread_start_us":72,"threads_started":1}
I20260928 14:52:11.844101 9082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:11.844492 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:11.844349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:38743 (local address 127.0.0.1:35228)
0928 14:52:11.844404 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:11.844461 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:52:11.844647 9030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
I20260928 14:52:11.845212 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:11.845121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:38743 (local address 127.0.0.1:35232)
0928 14:52:11.845151 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:11.845189 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 14:52:11.845304 9082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.189:38743: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.18.96.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab.
Entry for principal kudu/127.18.96.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab.
I20260928 14:52:11.867831 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.189:38743
--webserver_interface=127.18.96.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab
--principal=kudu/127.18.96.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:41071,127.18.96.189:38743,127.18.96.188:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:52:11.894541 9082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } attempt: 1
I20260928 14:52:11.895074 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:11.894926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:38743 (local address 127.0.0.1:35240)
0928 14:52:11.894995 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:11.895043 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 14:52:11.895493 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:11.895395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:38743 (local address 127.0.0.1:35244)
0928 14:52:11.895440 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:11.895466 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 14:52:11.895581 9082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.189:38743: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
W20260928 14:52:11.953052 9089 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:52:11.953325 9089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:11.953356 9089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:11.953400 9089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:11.954679 9089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:11.954761 9089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:11.954784 9089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:11.954803 9089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:11.954838 9089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:11.956437 9089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:41071,127.18.96.189:38743,127.18.96.188:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.189.keytab
--principal=kudu/127.18.96.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.189:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.96.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:11.956737 9089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:11.956988 9089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:11.962180 9097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:11.962180 9095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:11.962208 9094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:11.962596 9089 server_base.cc:1034] running on GCE node
I20260928 14:52:11.962994 9089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:11.963356 9089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:11.963701 9082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } attempt: 2
I20260928 14:52:11.964102 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:11.964005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:38743 (local address 127.0.0.1:35246)
0928 14:52:11.964043 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:11.964074 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 14:52:11.964414 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:11.964359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:38743 (local address 127.0.0.1:35260)
0928 14:52:11.964379 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:11.964401 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 14:52:11.964502 9082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.189:38743: Network error: Client connection negotiation failed: client connection to 127.18.96.189:38743: connect: Connection refused (error 111)
I20260928 14:52:11.964509 9089 hybrid_clock.cc:648] HybridClock initialized: now 1790607131964491 us; error 32 us; skew 500 ppm
Sep 28 14:52:11 dist-test-slave-j968 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:11.967331 9089 init.cc:377] Logged in from keytab as kudu/127.18.96.189@KRBTEST.COM (short username kudu)
I20260928 14:52:11.967829 9089 webserver.cc:460] Webserver started at http://127.18.96.189:44393/ using document root <none> and password file <none>
I20260928 14:52:11.968133 9089 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:11.968209 9089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:11.968384 9089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:11.969597 9089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/data/instance:
uuid: "6831331e451e4660b2b74bbd8ce295c0"
format_stamp: "Formatted at 2026-09-28 14:52:11 on dist-test-slave-j968"
server_key: "2a56eba13fb65c0ac96358c16b5f3696"
server_key_iv: "b3c71a591837b371c0dae788786ac5e8"
server_key_version: "encryptionkey@0"
I20260928 14:52:11.969987 9089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/wal/instance:
uuid: "6831331e451e4660b2b74bbd8ce295c0"
format_stamp: "Formatted at 2026-09-28 14:52:11 on dist-test-slave-j968"
server_key: "2a56eba13fb65c0ac96358c16b5f3696"
server_key_iv: "b3c71a591837b371c0dae788786ac5e8"
server_key_version: "encryptionkey@0"
I20260928 14:52:11.971499 9089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 14:52:11.972424 9104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:11.972643 9089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 14:52:11.972755 9089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/wal
uuid: "6831331e451e4660b2b74bbd8ce295c0"
format_stamp: "Formatted at 2026-09-28 14:52:11 on dist-test-slave-j968"
server_key: "2a56eba13fb65c0ac96358c16b5f3696"
server_key_iv: "b3c71a591837b371c0dae788786ac5e8"
server_key_version: "encryptionkey@0"
I20260928 14:52:11.972836 9089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:11.991566 9089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:11.993793 9089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:11.993973 9089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:11.998226 9089 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.189:38743
I20260928 14:52:11.998312 9156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.189:38743 every 8 connection(s)
I20260928 14:52:11.998657 9089 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/data/info.pb
I20260928 14:52:11.999747 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:12.001554 9157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:12.003249 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 9089
I20260928 14:52:12.003346 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-1/wal/instance
I20260928 14:52:12.003582 18818 external_mini_cluster.cc:1428] Setting key 007cc18b159c7620e34972eb41751cbc
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.188@KRBTEST.COM; defaulting to no policy
I20260928 14:52:12.014020 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.002434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.18.96.190:41071)
0928 14:52:12.002556 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.002559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.002750 (+ 191us) server_negotiation.cc:407] Connection header received
0928 14:52:12.003609 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.003610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.003654 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.003732 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.004843 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.005573 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.007232 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.007620 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.007705 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.010679 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.010681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.010681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.010690 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.012429 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.012859 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.012860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.012860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.012988 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.013267 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.013268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.013268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.013390 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.013472 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.013803 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.013891 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 14:52:12.014220 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.002345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:41071 (local address 127.0.0.1:60530)
0928 14:52:12.002608 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.002656 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:12.003445 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:12.003855 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:12.003859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:12.004337 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:12.004700 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.004706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.005700 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.005701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.007118 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.007122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.007737 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.007738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.007954 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.008697 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:12.008709 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:12.010570 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:12.012524 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.012532 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.012534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.012786 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.013061 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.013062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.013063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.013178 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.013463 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:12.013470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:12.013665 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 14:52:12.013783 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 14:52:12.013953 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.188@KRBTEST.COM" created.
I20260928 14:52:12.014842 9157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:52:12.023495 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.015237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.18.96.189:38743)
0928 14:52:12.015417 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.015422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.015449 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:52:12.015473 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.015476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.015491 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.015564 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.015933 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.016910 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.017936 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.018360 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.018428 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.020662 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.020669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.020670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.020676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.022259 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.022688 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.022690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.022690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.022745 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.022982 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.022982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.022982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.023113 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.023171 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.023347 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.023416 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260928 14:52:12.023630 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.015158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:38743 (local address 127.0.0.1:35274)
0928 14:52:12.015249 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.015262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:12.015309 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:12.015590 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:12.015591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:12.015767 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:12.015831 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.015833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.017006 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.017007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.017845 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.017848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.018446 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.018447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.018534 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.019000 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:12.019007 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:12.020563 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:12.022349 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.022350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.022351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.022597 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.022817 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.022819 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.022819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.022896 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.023180 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:12.023181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:12.023233 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:52:12.023324 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:52:12.023490 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":46}
I20260928 14:52:12.024392 9157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:12.024801 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.024667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58662)
0928 14:52:12.024720 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.024776 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:52:12.024947 9108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
I20260928 14:52:12.025400 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.025288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58670)
0928 14:52:12.025335 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.025368 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:52:12.025517 9157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:41709: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab.
Entry for principal kudu/127.18.96.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab.
I20260928 14:52:12.032200 18818 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklohQF_/build/release/bin/kudu
/tmp/dist-test-tasklohQF_/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.188:41709
--webserver_interface=127.18.96.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab
--principal=kudu/127.18.96.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.96.148:45427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.190:41071,127.18.96.189:38743,127.18.96.188:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:52:12.055672 9082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } attempt: 3
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:52:12.063551 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.056186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.18.96.189:38743)
0928 14:52:12.056246 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.056248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.056275 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:52:12.056390 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.056391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.056402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.056444 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.056791 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.057302 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.058088 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.058337 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.058424 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.060702 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.060704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.060705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.060713 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.062295 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.062699 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.062700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.062701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.062767 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.062980 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.062981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.062981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.063108 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.063182 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.063354 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.063448 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:52:12.063558 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.056099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:38743 (local address 127.0.0.1:35286)
0928 14:52:12.056216 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.056230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:12.056295 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:12.056457 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:12.056458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:12.056630 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:12.056699 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.056701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.057391 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.057393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.057991 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.057993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.058426 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.058427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.058507 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.058996 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:12.059001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:12.060584 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:12.062409 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.062411 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.062411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.062614 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.062835 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.062836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.062837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.062907 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.063184 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:12.063185 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:12.063240 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 14:52:12.063327 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:52:12.063448 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 14:52:12.064242 9082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:12.064620 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.064530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58674)
0928 14:52:12.064566 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.064598 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 14:52:12.064949 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.064850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58686)
0928 14:52:12.064894 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.064919 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 14:52:12.065037 9082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:41709: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
I20260928 14:52:12.074784 9157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } attempt: 1
I20260928 14:52:12.075438 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.075259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58688)
0928 14:52:12.075344 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.075398 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 14:52:12.075794 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.075699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58704)
0928 14:52:12.075745 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.075773 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:52:12.075948 9157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:41709: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
I20260928 14:52:12.096168 9082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } attempt: 1
I20260928 14:52:12.096731 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.096570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58718)
0928 14:52:12.096658 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.096700 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 14:52:12.097122 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.097032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58722)
0928 14:52:12.097074 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.097100 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 14:52:12.097239 9082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:41709: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
W20260928 14:52:12.116309 9164 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:52:12.116528 9164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:52:12.116546 9164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:52:12.116567 9164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:52:12.117769 9164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:52:12.117818 9164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:52:12.117833 9164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:52:12.117846 9164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:52:12.117864 9164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:52:12.119311 9164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.148:45427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.190:41071,127.18.96.189:38743,127.18.96.188:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/krb5kdc/kudu_127.18.96.188.keytab
--principal=kudu/127.18.96.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.188:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.96.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 13:47:06 UTC on ccycloud.quasar-fguyrz.root.comops.site
build id 4677365
I20260928 14:52:12.119515 9164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:52:12.119699 9164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:52:12.124819 9172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:12.124819 9170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:52:12.124822 9169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:52:12.125298 9164 server_base.cc:1034] running on GCE node
I20260928 14:52:12.125586 9164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:52:12.125906 9164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:52:12.127079 9164 hybrid_clock.cc:648] HybridClock initialized: now 1790607132127046 us; error 45 us; skew 500 ppm
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:52:12.130023 9164 init.cc:377] Logged in from keytab as kudu/127.18.96.188@KRBTEST.COM (short username kudu)
I20260928 14:52:12.130550 9164 webserver.cc:460] Webserver started at http://127.18.96.188:35379/ using document root <none> and password file <none>
I20260928 14:52:12.130774 9164 fs_manager.cc:359] Metadata directory not provided
I20260928 14:52:12.130820 9164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:52:12.130918 9164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:52:12.132347 9164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/data/instance:
uuid: "6beb423ecd6b487db50ad9d2ad811c12"
format_stamp: "Formatted at 2026-09-28 14:52:12 on dist-test-slave-j968"
server_key: "5117b348328a636e66028ee50526e946"
server_key_iv: "a6a04bba54b6b9b41e4d323237987dc8"
server_key_version: "encryptionkey@0"
I20260928 14:52:12.132853 9164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/wal/instance:
uuid: "6beb423ecd6b487db50ad9d2ad811c12"
format_stamp: "Formatted at 2026-09-28 14:52:12 on dist-test-slave-j968"
server_key: "5117b348328a636e66028ee50526e946"
server_key_iv: "a6a04bba54b6b9b41e4d323237987dc8"
server_key_version: "encryptionkey@0"
I20260928 14:52:12.136277 9164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:52:12.137449 9179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:52:12.137640 9164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:52:12.137748 9164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/data,/tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/wal
uuid: "6beb423ecd6b487db50ad9d2ad811c12"
format_stamp: "Formatted at 2026-09-28 14:52:12 on dist-test-slave-j968"
server_key: "5117b348328a636e66028ee50526e946"
server_key_iv: "a6a04bba54b6b9b41e4d323237987dc8"
server_key_version: "encryptionkey@0"
I20260928 14:52:12.137835 9164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:52:12.144232 9157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } attempt: 2
I20260928 14:52:12.144718 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.144576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58734)
0928 14:52:12.144641 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.144676 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:52:12.145069 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.144963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58744)
0928 14:52:12.145014 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.145042 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 14:52:12.145172 9157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.96.188:41709: Network error: Client connection negotiation failed: client connection to 127.18.96.188:41709: connect: Connection refused (error 111)
I20260928 14:52:12.164032 9164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:52:12.166342 9164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:52:12.166538 9164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:52:12.170893 9164 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.188:41709
I20260928 14:52:12.170970 9231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.96.188:41709 every 8 connection(s)
I20260928 14:52:12.171378 9164 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/data/info.pb
I20260928 14:52:12.172384 9082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } attempt: 2
I20260928 14:52:12.172587 9232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:52:12.174737 9232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:12.177428 18818 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklohQF_/build/release/bin/kudu as pid 9164
I20260928 14:52:12.177541 18818 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklohQF_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606857360589-18818-0/minicluster-data/master-2/wal/instance
I20260928 14:52:12.177795 18818 external_mini_cluster.cc:1428] Setting key 7b3d996218a049444c28a4cf2f0cc36c
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.190@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:52:12.184248 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.172736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58760)
0928 14:52:12.172786 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.172801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:12.172872 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:12.173971 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:12.173972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:12.174166 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:12.174224 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.174227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.175618 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.175620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.176784 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.176788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.177734 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.177736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.177808 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.178338 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:12.178343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:12.180088 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:12.182496 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.182497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.182497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.182864 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.183162 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.183163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.183163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.183229 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.183761 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:12.183762 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:12.183814 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:52:12.184020 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 14:52:12.184132 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 14:52:12.184476 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.175439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.18.96.190:41071)
0928 14:52:12.175509 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.175511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.175682 (+ 171us) server_negotiation.cc:407] Connection header received
0928 14:52:12.175752 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.175753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.175765 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.175819 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.176378 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.176963 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.178120 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.178464 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.178541 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.181191 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.181193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.181193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.181201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.183036 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.183492 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.183493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.183494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.183547 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.183798 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.183800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.183800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.183968 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.184028 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.184288 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.184369 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:52:12.184520 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.172901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.18.96.188:41709)
0928 14:52:12.173081 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.173090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.173122 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:52:12.173756 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.173763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.173816 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.173978 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.174350 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.175524 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.176875 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.177575 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.177764 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.180184 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.180191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.180192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.180199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.182363 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.182972 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.182976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.182977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.183093 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.183314 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.183315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.183315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.183676 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.183785 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.184108 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.184243 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:52:12.185352 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.175409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.190:41071 (local address 127.0.0.1:60534)
0928 14:52:12.175566 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.175624 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:12.175694 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:12.175836 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:12.175839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:12.176215 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:12.176287 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.176289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.177149 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.177151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.177985 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.177989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.178582 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.178584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.178658 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.179376 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:12.179392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:12.181065 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:12.183118 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.183126 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.183128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.183406 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.183628 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.183629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.183629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.183705 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.184085 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:12.184093 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:12.184145 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:52:12.185088 (+ 943us) client_negotiation.cc:240] Negotiation successful
0928 14:52:12.185218 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260928 14:52:12.185964 9232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:52:12.186487 9082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205: Bootstrap starting.
I20260928 14:52:12.187350 9082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:12.187688 9082 log.cc:824] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:12.188637 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.178725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.18.96.188:41709)
0928 14:52:12.178901 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.178903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.178911 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:12.178932 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.178933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.178945 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.179012 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.179927 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.180638 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.182182 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.182374 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.182459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.185202 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.185203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.185203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.185210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.187255 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.187650 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.187651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.187651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.187710 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.188016 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.188017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.188018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.188165 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.188297 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.188413 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.188499 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260928 14:52:12.188977 9082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205: No bootstrap required, opened a new log
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:52:12.191182 9082 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } }
I20260928 14:52:12.191344 9082 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf710f93892240bd87801b6024c62205, State: Initialized, Role: FOLLOWER
I20260928 14:52:12.191435 9082 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } }
I20260928 14:52:12.191799 9082 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:12.191776 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.192257 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:12.194370 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.186295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.189:38743 (local address 127.0.0.1:35288)
0928 14:52:12.186347 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.186357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:12.186392 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:12.186624 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:12.186625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:12.186777 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:12.186837 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.186839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.187893 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.187894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.188792 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.188794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.189314 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.189315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.189401 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.189869 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:12.189874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:12.191270 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:12.193150 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.193151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.193152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.193361 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.193621 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.193622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.193622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.193686 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.194009 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:12.194011 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:12.194059 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:52:12.194150 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:52:12.194260 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 14:52:12.194372 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.186378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.18.96.189:38743)
0928 14:52:12.186424 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.186426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.186435 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:52:12.186463 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.186464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.186474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.186536 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.186958 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.187754 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.188887 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.189138 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.189260 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.191345 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.191346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.191347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.191355 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.193062 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.193449 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.193451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.193451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.193522 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.193765 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.193766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.193766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.193920 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.194003 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.194192 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.194276 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.190@KRBTEST.COM
I20260928 14:52:12.195147 9232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } has no permanent_uuid. Determining permanent_uuid...
W20260928 14:52:12.195277 9250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:52:12.195322 9250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:52:12.196502 9014 master_runner.cc:186] Error getting master registration for 127.18.96.189:38743: OK, instance_id { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" instance_seqno: 1790607131996911 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:52:12.199735 9014 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:52:12.199817 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.189566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.18.96.190:41071)
0928 14:52:12.189603 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.189604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.189613 (+ 9us) server_negotiation.cc:407] Connection header received
0928 14:52:12.189701 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.189703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.189715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.189792 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.190385 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.191202 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.192658 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.192973 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.193020 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.195533 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.195534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.195534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.195542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.198112 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.198562 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.198563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.198564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.198639 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.198855 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.198856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.198856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.198991 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.199135 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.199568 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.199667 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607132, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.188@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:52:12.203202 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.195378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58772)
0928 14:52:12.195413 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.195424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:12.195463 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:12.195573 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:12.195574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:12.195719 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:12.195756 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.195758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.196753 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.196754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.197861 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.197864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.198371 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.198371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.198416 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.198799 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:12.198803 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:12.200370 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:12.202134 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.202135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.202136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.202306 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.202514 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.202515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.202516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.202569 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.202855 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:12.202856 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:12.202896 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 14:52:12.202965 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:52:12.203089 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":2}
I20260928 14:52:12.203244 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.195456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.18.96.188:41709)
0928 14:52:12.195487 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.195489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.195496 (+ 7us) server_negotiation.cc:407] Connection header received
0928 14:52:12.195515 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.195516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.195526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.195573 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.195827 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.196653 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.197959 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.198270 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.198326 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.200450 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.200452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.200452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.200458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.201919 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.202396 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.202397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.202397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.202451 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.202639 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.202640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.202640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.202788 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.202864 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.202990 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.203138 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":16}
I20260928 14:52:12.204877 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12: Bootstrap starting.
I20260928 14:52:12.205539 9232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:12.205839 9232 log.cc:824] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.189@KRBTEST.COM
I20260928 14:52:12.206611 9232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12: No bootstrap required, opened a new log
I20260928 14:52:12.209057 9232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } }
I20260928 14:52:12.209229 9232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6beb423ecd6b487db50ad9d2ad811c12, State: Initialized, Role: FOLLOWER
I20260928 14:52:12.209329 9232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } }
I20260928 14:52:12.209626 9253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.209692 9232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:12.209772 9253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:12.210461 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.200574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.18.96.189:38743)
0928 14:52:12.200618 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.200620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.200628 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:12.200651 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.200652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.200663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.200756 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.201416 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.202242 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.203372 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.203746 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.203794 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.206844 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.206846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.206846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.206854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.208747 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.209207 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.209208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.209209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.209372 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.209703 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.209704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.209704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.209871 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.210016 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.210256 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.210354 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 14:52:12.212072 9264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:52:12.212131 9264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:52:12.212390 18818 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:52:12.213495 9056 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bf710f93892240bd87801b6024c62205"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60550
I20260928 14:52:12.213690 9056 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:52:12.213752 9056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:12.214823 9056 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } }
I20260928 14:52:12.215067 9056 leader_election.cc:288] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [CANDIDATE]: Term 1 election: Requested vote from peers 6831331e451e4660b2b74bbd8ce295c0 (127.18.96.189:38743), 6beb423ecd6b487db50ad9d2ad811c12 (127.18.96.188:41709)
I20260928 14:52:12.213140 9164 master_runner.cc:186] Error getting master registration for 127.18.96.189:38743: OK, instance_id { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" instance_seqno: 1790607131996911 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:52:12.215220 9164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:52:12.215488 9206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf710f93892240bd87801b6024c62205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6beb423ecd6b487db50ad9d2ad811c12"
I20260928 14:52:12.215605 9206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:12.216666 9206 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf710f93892240bd87801b6024c62205 in term 1.
I20260928 14:52:12.216964 9033 leader_election.cc:302] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6beb423ecd6b487db50ad9d2ad811c12, bf710f93892240bd87801b6024c62205; no voters:
I20260928 14:52:12.217125 9239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:52:12.217319 9239 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [term 1 LEADER]: Becoming Leader. State: Replica: bf710f93892240bd87801b6024c62205, State: Running, Role: LEADER
I20260928 14:52:12.217446 9239 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } }
I20260928 14:52:12.218029 9273 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf710f93892240bd87801b6024c62205. Latest consensus state: current_term: 1 leader_uuid: "bf710f93892240bd87801b6024c62205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.218286 9273 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:12.218601 9275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:52:12.218796 9275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:52:12.222424 9206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [term 1 FOLLOWER]: Refusing update from remote peer bf710f93892240bd87801b6024c62205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:52:12.222865 9273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:52:12.225876 9253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf710f93892240bd87801b6024c62205. Latest consensus state: current_term: 1 leader_uuid: "bf710f93892240bd87801b6024c62205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.226009 9253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:12.226104 9276 mvcc.cc:205] Tried to move back new op lower bound from 7334326813577838592 to 7334326813563568128. Current Snapshot: MvccSnapshot[applied={T|T < 7334326813577838592}]
I20260928 14:52:12.226528 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf710f93892240bd87801b6024c62205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.226647 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [sys.catalog]: This master's current role is: LEADER
I20260928 14:52:12.227233 9253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf710f93892240bd87801b6024c62205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.227334 9253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:12.227411 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.216694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.18.96.189:38743)
0928 14:52:12.216740 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.216742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.218155 (+ 1413us) server_negotiation.cc:407] Connection header received
0928 14:52:12.218270 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.218271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.218285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.218319 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.219209 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.219844 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.221163 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.221512 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.221595 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.223364 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.223365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.223366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.223375 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.225436 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.225994 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.225996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.225996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.226070 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.226287 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.226288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.226289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.226434 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.227096 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.227141 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.227228 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 14:52:12.232430 9275 catalog_manager.cc:1347] Generated new cluster ID: 91dea0c56cb3442caba3fd8efa4d4dae
I20260928 14:52:12.232496 9275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:52:12.236294 9157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } attempt: 3
I20260928 14:52:12.239605 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.216988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.18.96.188:41709)
0928 14:52:12.217144 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.217146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.224631 (+ 7485us) server_negotiation.cc:407] Connection header received
0928 14:52:12.226565 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.226566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.226579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.226625 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.228335 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.229115 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.230162 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.230386 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.230477 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.232042 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.232044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.232044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.232051 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.237525 (+ 5474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.238163 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.238164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.238164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.238219 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.238510 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.238511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.238512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.238962 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.239076 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.239438 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.239523 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260928 14:52:12.240772 9277 mvcc.cc:205] Tried to move back new op lower bound from 7334326813577838592 to 7334326813563568128. Current Snapshot: MvccSnapshot[applied={T|T < 7334326813577838592}]
I20260928 14:52:12.241575 9275 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:52:12.242470 9275 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 14:52:12 dist-test-slave-j968 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607131, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.189@KRBTEST.COM for kudu/127.18.96.188@KRBTEST.COM
I20260928 14:52:12.245656 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.216741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.18.96.190:41071)
0928 14:52:12.216779 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.216781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.226273 (+ 9492us) server_negotiation.cc:407] Connection header received
0928 14:52:12.226447 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.226448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.226459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.226493 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.228664 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.229337 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.237847 (+ 8510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.238174 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.238222 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.240199 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.240201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.240201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.240209 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.242092 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.244104 (+ 2012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.244105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.244105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.244167 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.244751 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.244752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.244752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.244896 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.244994 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.245435 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.245541 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 14:52:12.247054 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.236632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.96.188:41709 (local address 127.0.0.1:58792)
0928 14:52:12.236734 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 14:52:12.236745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:52:12.236789 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:52:12.237442 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:52:12.237443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:52:12.237633 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:52:12.237683 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.237686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.238665 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.238667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.239601 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:52:12.239606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.240287 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.240288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:52:12.240345 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.240928 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:52:12.240933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:52:12.243357 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:52:12.245540 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.245541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.245541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.245760 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.246307 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:52:12.246307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:52:12.246308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:52:12.246377 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:52:12.246700 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:52:12.246701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:52:12.246800 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 14:52:12.246864 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:52:12.246967 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 14:52:12.247567 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:12.237225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.18.96.188:41709)
0928 14:52:12.237278 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.237280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.237290 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:52:12.237313 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.237314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.237325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.237387 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.237775 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.238565 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.239789 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:12.240130 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:12.240230 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:12.243466 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:52:12.243468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:52:12.243469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:52:12.243476 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:52:12.245437 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.246151 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.246152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.246152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.246223 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:52:12.246454 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:52:12.246455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:52:12.246455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:52:12.246589 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:52:12.246651 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:12.247383 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 14:52:12.247469 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:52:12.247640 9275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205: Generated new TSK 0
I20260928 14:52:12.247921 9275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:52:12.249456 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0: Bootstrap starting.
I20260928 14:52:12.250242 9157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0: Neither blocks nor log segments found. Creating new log.
I20260928 14:52:12.250540 9157 log.cc:824] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:52:12.251484 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0: No bootstrap required, opened a new log
I20260928 14:52:12.253818 9157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } }
I20260928 14:52:12.253970 9157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6831331e451e4660b2b74bbd8ce295c0, State: Initialized, Role: FOLLOWER
I20260928 14:52:12.254050 9157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } }
I20260928 14:52:12.254397 9157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:52:12.256075 9130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:52:12.255774 9131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf710f93892240bd87801b6024c62205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6831331e451e4660b2b74bbd8ce295c0"
I20260928 14:52:12.257061 9130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [term 1 FOLLOWER]: Refusing update from remote peer bf710f93892240bd87801b6024c62205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:52:12.257365 9239 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:52:12.257933 9131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf710f93892240bd87801b6024c62205 in term 1.
I20260928 14:52:12.261011 9297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf710f93892240bd87801b6024c62205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.261137 9297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:12.261204 9297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf710f93892240bd87801b6024c62205. Latest consensus state: current_term: 1 leader_uuid: "bf710f93892240bd87801b6024c62205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.261256 9297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:52:12.261296 9297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf710f93892240bd87801b6024c62205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.261344 9297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:52:12.268774 9296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:52:12.268864 9296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 14:52:12.276103 9234 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.273226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.18.96.188:41709)
0928 14:52:12.273277 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.273280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.274416 (+ 1136us) server_negotiation.cc:407] Connection header received
0928 14:52:12.275411 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.275413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.275427 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.275461 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.276025 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58800: BlockingRecv error: recv got EOF from 127.0.0.1:58800 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 14:52:12.277036 9085 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.273675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.18.96.190:41071)
0928 14:52:12.273719 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.273721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.276418 (+ 2697us) server_negotiation.cc:407] Connection header received
0928 14:52:12.276528 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.276529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.276542 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.276574 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.276971 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60578: BlockingRecv error: recv got EOF from 127.0.0.1:60578 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:52:12.277822 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.274061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.18.96.189:38743)
0928 14:52:12.274111 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.274113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.277286 (+ 3173us) server_negotiation.cc:407] Connection header received
0928 14:52:12.277390 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.277391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.277403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.277435 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.277762 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35318: BlockingRecv error: recv got EOF from 127.0.0.1:35318 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 14:52:12.280531 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.279644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.18.96.189:38743)
0928 14:52:12.279683 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.279685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.279967 (+ 282us) server_negotiation.cc:407] Connection header received
0928 14:52:12.280094 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.280095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.280114 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.280147 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.280482 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35328: BlockingRecv error: recv got EOF from 127.0.0.1:35328 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 14:52:12.283816 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.282995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.18.96.189:38743)
0928 14:52:12.283030 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.283032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.283279 (+ 247us) server_negotiation.cc:407] Connection header received
0928 14:52:12.283385 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.283386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.283397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.283429 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.283770 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35340: BlockingRecv error: recv got EOF from 127.0.0.1:35340 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 14:52:12.288200 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.287553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.18.96.189:38743)
0928 14:52:12.287588 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.287590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.287697 (+ 107us) server_negotiation.cc:407] Connection header received
0928 14:52:12.287790 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.287791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.287800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.287828 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.288157 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35352: BlockingRecv error: recv got EOF from 127.0.0.1:35352 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 14:52:12.297116 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.296188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.18.96.189:38743)
0928 14:52:12.296237 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.296239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.296513 (+ 274us) server_negotiation.cc:407] Connection header received
0928 14:52:12.296620 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.296621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.296632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.296665 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.297053 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35364: BlockingRecv error: recv got EOF from 127.0.0.1:35364 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 14:52:12.309329 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.308334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.18.96.189:38743)
0928 14:52:12.308381 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.308383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.308659 (+ 276us) server_negotiation.cc:407] Connection header received
0928 14:52:12.308782 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.308784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.308823 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.308863 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.309253 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35366: BlockingRecv error: recv got EOF from 127.0.0.1:35366 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 14:52:12.327688 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.326757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.18.96.189:38743)
0928 14:52:12.326803 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.326805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.327066 (+ 261us) server_negotiation.cc:407] Connection header received
0928 14:52:12.327176 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.327177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.327192 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.327225 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.327609 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35382: BlockingRecv error: recv got EOF from 127.0.0.1:35382 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:52:12.362166 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.361186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.18.96.189:38743)
0928 14:52:12.361233 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.361236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.361497 (+ 261us) server_negotiation.cc:407] Connection header received
0928 14:52:12.361607 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.361608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.361621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.361653 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.362103 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35388: BlockingRecv error: recv got EOF from 127.0.0.1:35388 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 14:52:12.428473 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.427627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.18.96.189:38743)
0928 14:52:12.427679 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.427682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.427821 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:52:12.427944 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.427945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.427958 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.428034 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.428412 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35400: BlockingRecv error: recv got EOF from 127.0.0.1:35400 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 14:52:12.558387 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.557551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.18.96.189:38743)
0928 14:52:12.557636 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.557638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.557770 (+ 132us) server_negotiation.cc:407] Connection header received
0928 14:52:12.557875 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.557876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.557888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.557965 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.558319 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35414: BlockingRecv error: recv got EOF from 127.0.0.1:35414 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:52:12.647059 9280 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf710f93892240bd87801b6024c62205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf710f93892240bd87801b6024c62205" member_type: VOTER last_known_addr { host: "127.18.96.190" port: 41071 } } peers { permanent_uuid: "6831331e451e4660b2b74bbd8ce295c0" member_type: VOTER last_known_addr { host: "127.18.96.189" port: 38743 } } peers { permanent_uuid: "6beb423ecd6b487db50ad9d2ad811c12" member_type: VOTER last_known_addr { host: "127.18.96.188" port: 41709 } } }
I20260928 14:52:12.647239 9280 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf710f93892240bd87801b6024c62205 [sys.catalog]: This master's current role is: LEADER
W20260928 14:52:12.816457 9161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:52:12.815777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.18.96.189:38743)
0928 14:52:12.815850 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:12.815852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:12.815860 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:12.815906 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:12.815907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:12.815917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:12.815978 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:52:12.816321 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35430: BlockingRecv error: recv got EOF from 127.0.0.1:35430 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:52:13.212585 9264 catalog_manager.cc:1259] Loaded cluster ID: 91dea0c56cb3442caba3fd8efa4d4dae
I20260928 14:52:13.212666 9264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12: loading cluster ID for follower catalog manager: success
I20260928 14:52:13.213551 9264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12: acquiring CA information for follower catalog manager: success
I20260928 14:52:13.213954 9264 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6beb423ecd6b487db50ad9d2ad811c12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:52:13.269361 9296 catalog_manager.cc:1259] Loaded cluster ID: 91dea0c56cb3442caba3fd8efa4d4dae
I20260928 14:52:13.269445 9296 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0: loading cluster ID for follower catalog manager: success
I20260928 14:52:13.270450 9296 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0: acquiring CA information for follower catalog manager: success
I20260928 14:52:13.270829 9296 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6831331e451e4660b2b74bbd8ce295c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:52:13.332589 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:13.329714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.18.96.189:38743)
0928 14:52:13.329820 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:13.329822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:13.329858 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:52:13.329884 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:13.329885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:13.329938 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:13.330007 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:13.330254 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.330750 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.331590 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.331780 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.331830 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:13.332015 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:13.332158 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:13.332345 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:13.332424 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:52:13.332449 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
I20260928 14:52:13.337293 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:13.334774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.18.96.188:41709)
0928 14:52:13.334909 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:13.334910 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:13.334918 (+ 8us) server_negotiation.cc:407] Connection header received
0928 14:52:13.334958 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:13.334958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:13.334995 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:13.335067 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:13.335213 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.335696 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.336565 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.336778 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.336832 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:13.336985 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:13.337092 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:13.337148 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:13.337207 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 14:52:13.337225 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260928 14:52:13.342425 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:13.339373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.18.96.189:38743)
0928 14:52:13.339440 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:13.339442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:13.339741 (+ 299us) server_negotiation.cc:407] Connection header received
0928 14:52:13.339826 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:13.339827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:13.339839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:13.339930 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:13.340056 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.340769 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.341521 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.341816 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.341882 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:13.342058 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:13.342093 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:13.342144 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:13.342276 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 14:52:13.342303 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:52:13.342557 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:13.339255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.18.96.188:41709)
0928 14:52:13.339298 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:13.339300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:13.339448 (+ 148us) server_negotiation.cc:407] Connection header received
0928 14:52:13.339532 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:13.339533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:13.339541 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:13.339614 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:13.339773 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.340555 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.341592 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.341884 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.341942 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:13.342097 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:13.342153 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:13.342207 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:13.342444 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 14:52:13.342471 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:52:13.343209 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:52:13.339292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.18.96.190:41071)
0928 14:52:13.339420 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:52:13.339422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 14:52:13.339606 (+ 184us) server_negotiation.cc:407] Connection header received
0928 14:52:13.339682 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:52:13.339683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:52:13.339745 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:52:13.339827 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:52:13.340007 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.340518 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.341996 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:52:13.342345 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:52:13.342411 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:52:13.342777 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:52:13.342904 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:52:13.342968 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:52:13.343050 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 14:52:13.343081 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260928 14:52:13.345024 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 9014
I20260928 14:52:13.350524 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 9089
I20260928 14:52:13.356020 18818 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklohQF_/build/release/bin/kudu with pid 9164
2026-09-28T14:52:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2828 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5706 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (276002 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST